Translational Assessments of Reward and Anhedonia: A Tribute to Athina Markou

Abstract

Loss of pleasure (clinically referred to as anhedonia), impairments in other reward-related processes such as reward learning, motivation, and reward valuation, and blunted affect characterize several mood and other psychiatric disorders. Despite the availability of many therapeutic options for these disorders, reward-related impairments remain challenging to treat and often persist despite alleviation of other symptoms. Lack of animal models of reward-related impairments and affect that have high construct and predictive validity is a key obstacle to developing novel treatments. This review will highlight: 1) guidelines to consider when developing translatable animal models; and 2) recent efforts to develop new reward-related assessments in humans and non-human animals that have been translated or back-translated from one species to another. The procedures described in this review are used to assess aspects of reward learning, motivated behavior, reward valuation, and affect. In several cases, researchers have attempted to implement task parameters that are as identical as possible to the parallel parameters used in existing cross-species tasks, with the goal of improving the translation of preclinical drug discovery findings to the clinic. In this regard, Dr. Athina Markou had great influence on conceptualizing the development and use of translational animal models of reward-related processes. Thus, this review is dedicated to Dr. Markou and her tireless efforts throughout her career to understand and treat reward-related impairments across several psychiatric disorders.

Introduction

Loss of interest or pleasure (i.e., anhedonia) and other reward-related impairments characterize several psychiatric and neurological disorders (1, 2), including major depressive disorder (MDD) (3), bipolar disorder (4), schizophrenia (5, 6), posttraumatic stress disorder (7), and substance use disorder (particularly during withdrawal) (8). Despite the high prevalence of reward-related impairments across disorders, there are no approved medications to treat these debilitating symptoms. This is concerning since: (i) first-line antidepressant pharmacological (e.g., selective serotonin reuptake inhibitors; SSRI) and psychological (e.g., cognitive behavior therapy) treatments often fail to restore hedonic tone (9, 10), and (ii) anhedonia and reward-related dysfunctions predict poor treatment outcome, chronicity, and increased relapse risk (11–13).

While several reasons account for this lack of progress in treating reward-related impairments, we will focus here on the role of animal models of different reward processes in bridging the gap between preclinical discovery and treatment. We start by emphasizing guidelines for developing translational behavioral assessments that can be fruitfully used across species. Next, consistent with mounting evidence highlighting distinct subdomains of reward processing, we review translational tasks that have been developed for parallel use in humans and non-human animals to probe reward learning, motivation, reward valuation, and affect. We conclude by highlighting limitations of current work and future directions, including the utility to: (i) implement computational modeling to formally probe sub-processes underlying task performance (e.g., 14); and (ii) assess behavior in conjunction with physiological recordings (e.g., EEG) to more directly evaluate cross-species confluence.

Much of the conceptual and methodological points emphasized here were inspired by the seminal work of Dr. Athina Markou. With her uncompromising dedication to translational research, methodological rigor, and conceptual sophistication, Dr. Markou profoundly shaped the work of many basic and clinical scientists, including the authors of this review. We are indebted to her for her guidance, mentorship, countless discussions, and good humor, which made working with her a privilege. This review is dedicated to her and her pioneering contributions to translational research, which have fundamentally contributed to a better understanding of the pathophysiology of psychiatric and substance disorders, and the development of better treatments.

Prelude

It is important to define the term “model” when referring to non-human animal behavioral assessments related to psychiatric disorders. Such measures often assess a single, specific behavior that may or may not have good construct or predictive validity, yet psychiatric disorders include a much broader array of clinical symptoms, some of which are impossible to replicate in non-human animals. The National Institute of Mental Health (NIMH) Research Domain Criteria (RDoC) initiative, which aims to classify mental disorders based on specific behavioral dimensions, promotes the identification and treatment of specific behavioral symptoms (15). Thus, the term model used here refers to non-human animal assessments of specific behaviors linked to parallel human behaviors and symptoms, rather than entire psychiatric syndromes.

Guidelines for Developing Cross-species Translational Behavioral Assessments

Recent attempts to develop assessments of human reward-related behaviors in non-human animals have followed a relatively novel strategy: develop a preclinical version of an existing clinical assessment. In order for this cross-species approach to be successful, a few guidelines should be considered:

1. Anhedonia is frequently assessed in humans using self-report questionnaires (e.g., Snaith-Hamilton Pleasure Scale (16)) that are subjective and cannot be replicated in non-human animals. Moreover, with few exceptions (e.g., Dimensional Anhedonia Rating Scale (17)), scales probe single domains of anhedonia (e.g., consummatory pleasure). Accordingly, clinical assessments should minimize verbal communication, other than basic pre-test instructions. Even simple instructions may take months of training in rodents depending on task complexity. Thus, researchers should consider the extent of training required in non-human animals to approximate a human participant prepared to perform a task with brief instructions.

2. Task parameters should be identical, or as similar as possible, across species. Parameters to consider include number of trials and timing of stimuli and inter-trial intervals. Additionally, operant-based tasks typically utilize visual or auditory stimuli, which can be identical across several parameters (e.g., intensity, duration, inter-stimulus intervals) between species. One caveat is that reinforcers are difficult to match across species. For example, humans typically receive monetary rewards (i.e., extrinsic reinforcers), whereas non-human animals typically receive food or other palatable rewards (i.e., intrinsic reinforcers). Critically, whereas studies have shown that monetary and food rewards recruit a common set of brain regions (e.g., ventral striatum, amygdala, ventromedial prefrontal cortex), direct comparisons also highlighted important differences (18). Specifically, relative to food, monetary reward elicited stronger activation in the ventral striatum and evolutionary newer regions of the anterior orbitofrontal cortex (OFC), whereas food recruited more strongly the anterior insula and phylogenetically older regions in the posterior OFC relative to money. Moreover, whereas food deprivation is tightly regulated in non-human animals to facilitate behavioral responding, responding for extrinsic reinforcers in humans is likely more variable and involves factors beyond the experimenter’s control (e.g., attitudes toward money).

3. Responses should be similar across species (e.g., lever press in rodents vs. keyboard press in humans for operant-based tasks). Importantly, human responses should be non-verbal and objectively measured to best mimic preclinical studies.

4. Wherever possible, statistical analyses should be identical across species. This can be facilitated by increasing correspondence in task parameters between cross-species assessments.

5. Combining behavioral assessments with biological/physiological signals during testing will greatly strengthen the validity of translational tasks. There are considerable challenges to this approach. Notably, clinical researchers cannot use many of the invasive neurophysiological techniques utilized by preclinical researchers. Conversely, imaging techniques (e.g., EEG, fMRI) that require little/no movement during testing can be challenging at best for researchers working with non-human animals. Nonetheless, demonstration that similar neural or other relevant biological changes accompany behavioral changes will greatly enhance the translational value of cross-species behavioral tasks.

6. Cross-species behavioral assessments should be validated using manipulations that are analogous across species. This can most readily be achieved using pharmacological agents. Investigators should carefully consider the equivalence of doses and pretreatment times across species, which may be determined by examining the pharmacokinetic properties of test compounds in different species. Importantly, such properties can be altered by different routes of administration, which often vary across species. Even with identical routes of administration, important confounds should be considered. For example, oral administration that is relatively trivial in humans may be aversive in rodents using gavage.

Given these guidelines, it is important to consider the limitations of relying solely on face validity when developing cross-species tasks. For example, humans excel at discriminating visual cues, whereas rodents have poor visual acuity and are better at recognizing olfactory cues. If the ultimate goal of the task(s) is to assess reward functioning, it may be advantageous to use different stimuli based on each species’ most acute sensory modality. As described above, concurrent neurophysiological assessment will help identify whether humans and non-human animals are similarly engaged in their respective tasks, despite differences in task parameters.

The goal of this review is not to describe non-human animal assessments of reward that map onto human reward constructs (for several reviews on this, see 19, 20–22). Nor is the goal to identify reward constructs impaired in psychiatric disorders based on how different clinical populations respond in the assessments described below. Rather, we will focus on recent developments of human and non-human animal assessments designed to be analogous. The following translational behavioral assessments have been developed (and some have been systematically validated) for use in humans and primarily rodents using some of the guidelines described above. They include tasks relevant to reward learning, motivation, valuation, and affect.

Translational Assessments of Probabilistic Reward Learning

Probabilistic reward learning requires determination of the probability that a behavioral response will result in a rewarding outcome, then adapting behavior to maximize future rewards. Although probabilistic reward learning involves aspects of cognition (i.e., associative learning), responsiveness to rewards is a key feature that makes these tasks valuable for assessing reward-related impairments. Several variations of probabilistic reward learning tasks have been developed for use in both humans and non-human animals.

Probabilistic Learning Task

The Probabilistic Learning Task (PLT) is an assessment of learning associated with both positive and negative feedback (23, 24). Two stimuli are presented and subjects must respond for the “target” stimulus. Target responses are reinforced on a probabilistic schedule (e.g., 80% reinforcement rate). Similarly, non-target responses are not reinforced on a probabilistic schedule (80%). Thus, both target and non-target responses result in misleading feedback (i.e., no reward or reward, respectively) on 20% of trials, which elicits negative and positive reward prediction errors (RPE), respectively, that have been closely linked to decreases and increases, respectively, of firing in striatal and midbrain dopaminergic neurons (25). Because healthy subjects are expected to ignore misleading feedback, two behavioral measures of interest in the PLT are win-stay behavior (i.e., repeating a previously rewarded response) and lose-shift behavior (i.e., not repeating a previously non-rewarded response).

In the human PLT, the two stimuli may be different characters or shapes presented on a computer screen (26). Participants use a keyboard to indicate the target stimulus. Reward feedback is typically a confirmatory (e.g., “Correct!”) or monetary message. In the rodent PLT, subjects typically perform operant responses (e.g., nose poke or lever press) in the absence of additional stimuli (27). Advances in touchscreen technology allow subjects to respond to different stimuli on the screen, making the tasks more similar to the human versions (28). Rodents often respond for a food pellet or other palatable reward. More complex variations of the PLT have been developed, such as the Probabilistic Selection Task, but non-human versions of this task (29) differ considerably from human versions (30, 31).

Although there are human and non-human versions of the PLT, there is little direct comparison of task performance across species using similar manipulations. In healthy humans (i.e., without a psychiatric diagnosis), a low dose of the SSRI citalopram (30 mg), expected to decrease forebrain serotonin, increased lose-shift behavior (32). Similarly, in healthy rats, a low dose of citalopram (1 mg/kg) also increased lose-shift behavior (27), although only during the reversal phase of the task (see below). Moreover, higher doses of citalopram (expected to increase forebrain serotonin) decreased lose-shift behavior in rats. These findings highlight the importance of accurately determining comparable doses across species for pharmacological manipulations in translational tasks.

Probabilistic Reversal Learning Task

A common variant of the PLT is the Probabilistic Reversal Learning Task (PRL), which is used to assess cognitive flexibility based on rewards. Initially, the PRL is identical to the PLT – subjects learn to associate different stimuli with high (80%) or low (20%) probabilities of reward. During this initial discrimination phase, subjects must respond consecutively for the target reward, regardless of feedback. When successful, the original non-target stimulus becomes the new target stimulus, and subjects must switch to responding for the new target stimulus. The target contingency continues to shift between stimuli after each response criterion is achieved, and one key measure is number of reversals between stimuli during a single test.

As described above, pharmacologically decreasing serotonin levels increased lose-shift behavior in the rat PRL (27). Similarly, healthy humans with allelic variation of the serotonin transporter gene expected to decrease extrasynaptic serotonin also showed increased lose-shift behavior (33). With regard to dopamine, administration of a dopamine D2 agonist impaired reversal learning in humans (34), while administration of a dopamine D3 agonist impaired reversal learning in rats (35). However, discrepancies in the literature also exist. Different serotonergic manipulations have mixed effects in healthy non-human primates (i.e., impaired reversals) (36) and humans (i.e., no effect) (33), while inhibition of dopamine transporters with different pharmacological compounds can impair and enhance reversal learning in humans (37) and mice (38), respectively. These latter findings highlight the importance of consistency in experimental manipulations when comparing behavior across species.

Probabilistic Reward Task

Like the PLT, the Probabilistic Reward Task (PRT) measures behavioral changes based on previous experiences with rewarding outcomes. The PRT includes a signal detection component where subjects correctly discriminate two stimuli to receive a reward. However, unlike the PLT, the two stimuli are difficult to distinguish. Correct identification of either stimulus is probabilistically reinforced (60% for one stimulus – “rich”, 20% for the other – “lean”). Because the stimuli are ambiguous and positive feedback is infrequent, feedback can be ambiguous as well. Thus, the probabilistic reinforcement schedule is concealed more in the PRT than the PLT.

In the human PRT developed by Pizzagalli and colleagues (39, modified after 40), the stimuli are short or long mouths on a schematic face on a computer screen and responses are made on a keyboard. Correct identifications of rich and lean stimuli are reinforced with monetary feedback on 60% and 20% of trials, respectively. In the rodent PRT developed by Markou and colleagues (41) and others (42), the stimuli are short or long auditory tones presented in an operant box and responses are made with a lever press. Reinforcement probabilities are identical to the human PRT, and rats receive a food pellet reward. Measures of task performance are calculated identically between species. A similar rat PRT was also recently developed using ambiguous odor cues (43) based on an analogous task developed for monkeys (44, 45).

In the PRT, healthy humans and rats develop a response bias for the rich stimulus, reflecting sensitivity to the differential reinforcement schedules (39, 41). Humans with current or past MDD and bipolar disorder, unaffected relatives of individuals with MDD, and those with high trait levels of anhedonia, develop a blunted response bias relative to controls (39, 46–49). Notably, such dysfunctions are particularly prominent in MDD subjects reporting anhedonia (12) or meeting criteria for the melancholic subtype of MDD (50), and specifically correlate with current and predict future anhedonic symptoms (39, 46, 51). Response bias is also blunted in humans and rats exposed to stress (51–54), withdrawing from chronic nicotine (55), and after administration of a low dose of the dopamine D2/D3 agonist pramipexole, which is expected to decrease dopaminergic signaling via activation of inhibitory autoreceptors (41, 56). Conversely, response bias is potentiated in humans and rats after acute administration of psychostimulants, which are expected to increase dopaminergic signaling (41, 57). Highlighting some specificity, blunted response bias has generally not emerged in samples with schizophrenia (58, 59), who have been found to be impaired in reinforcement learning tasks requiring explicit representations about reward associations (58).

Translational Assessments of Motivated Behaviors

Motivation is the desire to act or accomplish goals. Avolition, or impaired motivation, may contribute to other behavioral symptoms like social withdrawal and cognitive impairment (60) and can disrupt functional outcome and quality of life (61, 62). Recent human laboratory assessments of motivation described below are based closely on existing rodent tasks.

Progressive Ratio Test

In the progressive ratio (PR) test, to obtain a reward, subjects perform an operant response, which becomes exponentially more difficult for subsequent rewards until the subject stops responding. The final ratio completed to earn the last reward is the breakpoint and is interpreted as the maximum effort to earn a reward. Thus, decreased breakpoints reflect avolition (63). Task difficulty can be altered by manipulating the exponential response requirement.

In the rodent PR test, animals typically press a lever or nose-poke) to receive a palatable reward (63). Rodents stop responding to either: 1) collect a reward once the response requirement is reached; or 2) give up if the response requirement is too high (i.e., breakpoint). In some human PR tests, participants respond on a keyboard or manipulate a joystick to obtain a monetary reward (64, 65). As in rodents, the response requirement to obtain a reward is exponentially increased. However, in other human PR tests, before each trial, the response requirement is displayed and participants may choose to forgo the trial, in which case the next trial is initiated with a relatively lower response requirement (66). One advantage of this design is that motivation thresholds can be measured throughout the session, as opposed to the end. Additionally, participants decide before a trial whether or not to exert effort for the reward. Several confounding factors may affect breakpoints in the PR tests where response requirements increase sequentially, like satiety and physical stamina, which are less likely to affect breakpoints in the latter human PR tests described above since high effort options may be encountered early during the task. Nonetheless, factors such as income or attitude towards money may produce satiety in humans expected to show motivation toward relatively nominal monetary rewards.

People with MDD, bipolar disorder (tested during the depressed phase), and schizophrenia all have reduced breakpoints relative to healthy controls in human PR tests (64, 66, 67). Similarly, congenitally learned helpless rats, a genetic rodent model of depression, showed reduced breakpoints in a rat PR test (68). Stress, which precipitates symptoms of psychiatric disorders, and chronic corticosterone treatment also reduce breakpoints in a PR test (69, 70), although other studies found no effect of various chronic stressors on breakpoints (71–73). Greater consistency in task parameters (including PR schedules), conceptual aspects of motivation (e.g., before vs. after task performance), and experimental manipulations may help clarify these discrepancies.

Effort-related Choice

Effort-related choice (ERC) tasks probe decision making aspects of motivated behaviors. Subjects can obtain a small reward by exerting minimal effort or a larger reward by exerting greater effort. The effort required to obtain the larger reward is varied throughout the task, allowing experimenters to probe whether and how much effort will be exerted to obtain a larger reward.

In humans, ERC can be assessed using the Effort-Expenditure for Rewards Task (EEfRT), which was developed by Treadway and colleagues based on the rodent ERC task described below (see 74 for a summary and psychometric evaluation of four additional effort-based decision making tasks). Participants initially choose to perform a difficult or easy task (e.g., performing many key presses using a non-dominant finger vs. few key presses using a dominant finger, respectively). Difficult and easy task completion results in high and low monetary rewards, respectively, and the probability of receiving that reward is indicated prior to choosing task difficulty.

EEfRT is based on a rodent ERC task designed by Salamone and colleagues (75, 76). Like humans, rats choose to perform an easy or difficult task (e.g., pressing a lever few or many times or climbing a tall barrier). The easy option produces a small reward (e.g., one food pellet), whereas the difficult option produces a larger reward (e.g., four food pellets). The intensity of the difficult task (e.g., lever presses or barrier height) can be manipulated throughout the test. Preference for low over high effort/reward options is interpreted as avolition.

In humans, several psychiatric disorders, such as MDD, schizophrenia, and autism, are associated with reduced selections of the high reward/effort option (77–82). Additionally, the likelihood of choosing the difficult task negatively correlates with self-reported anhedonia when the probability of receiving a reward is high (83). In rats, restraint stress impairs ERC (71, 84). Interestingly, amphetamine, which elevates striatal dopamine (85), increased preference for the high effort/reward option in humans (86) and rats (87). These findings are consistent with evidence that decreased dopamine function impairs ERC in rats (75, 76).

Translational Assessments of Reward Valuation

Assessing relative value of rewards is closely linked to other aspects of reward processing like reward learning and motivation. For example, greater reward valuation may justify increased effort expenditure to obtain the reward and is useful for calculating cost/benefit ratios. Additionally, rewards with high probabilities of attainment contribute to valuation of future rewards.

Outcome Devaluation Task

The Outcome Devaluation Task (ODT) quantifies reward valuation. During the task, different operant responses result in different rewarding outcomes. Devaluing one of the expected outcomes typically increases responding for the non-devalued reward. Reward devaluation is accomplished by overexposure to the reward or by pairing it with a noxious stimulus.

In the human ODT, the rewarding stimuli may be qualitatively similar or different (e.g., food and money) (88–90). Participants respond on a keyboard by indicating preference for one of the two stimuli. The preferred stimulus is then devalued. For example, if the stimulus is food, participants may be instructed to eat prior to testing. Once sated, participants typically respond less for the food stimulus compared to the other stimulus. In the rodent ODT, subjects perform two operant responses (e.g., left and right lever press) to receive either of two stimuli (e.g., food vs. sucrose pellet). Prior to testing, one stimulus is devalued (e.g. free-feeding with food), increasing responding for sucrose pellets. One caveat is that in humans, prior negative experiences with the rewards (e.g., chronic food deprivation due to poverty) may impact how devaluation of those rewards affects responding and should be accounted for.

While few studies have directly compared the ODT across species, some suggest that stress (a socially evaluated cold pressor test in humans (91) and chronic unpredictable stress in rats (92)) impairs sensitivity to reward devaluation in both species, reflected by lack of a post-conditioning decrease in responding for the devalued reinforcer. Additionally, evidence from humans and rats suggests involvement of medial prefrontal cortex, orbitofrontal cortex, and striatum in outcome devaluation (93).

Translational Assessments of Affect

Affective Tone Discrimination Task

The Affective Tone Discrimination Task (ATDT) assesses negative biases in emotional processing (94). Like the PRT, the ATDT includes a signal detection component where subjects identify different stimuli. However, unlike the PRT, correct identification of one stimulus in the ATDT is rewarded, whereas correct identification of the other stimulus prevents punishment. Additionally, the outcomes are certain, not probabilistic. Thus, given an ambiguous stimulus (i.e., qualitatively similar to both reward- and punishment-associated stimuli), subjects typically respond half the time to obtain a reward and half to avoid punishment. Manipulations expected to improve affect (e.g., lithium treatment) increase responding on the reward-associated apparatus given the ambiguous stimulus (95). Conversely, factors expected to reduce affect (e.g., increased noradrenergic and glucocorticoid signaling mimicking physiological stress responses) decrease responding on the reward-associated apparatus (96, 97).

In the human ATDT, high and low frequency tones signal rewarding and aversive stimuli (counterbalanced), and responses are made on a keyboard. Correct responses are either reinforced with a monetary reward or prevent punishment (e.g., an aversive sound) (98). In the rodent ATDT, high and low frequency tones also signal reward and punishment and responses are made with a lever press. The reward is typically a sweetened solution or food pellet, whereas punishment is delivered by electric shock to the grid floor (95–97).

In humans, greater anxiety correlates with bias toward the tone associated with punishment (98). However, studies in humans using the ATDT to compare findings in rodents using the pharmacological manipulations described above are lacking. One caveat associated with this task is that individuals with schizophrenia (99), bipolar disorder (100), or Parkinson’s disease (101) show impaired discrimination of auditory cues. Such sensory/perceptual deficits should be taken into consideration as they may confound the interpretation of studies using the ATDT.

Conclusions

In this review, we summarized several behavioral procedures to assess reward-related processes impaired in psychiatric disorders, including reward learning, motivation, reward valuation, and affect. While these processes do not encompass all reward-related impairments in psychiatric disorders, we focused on those for which there has been recent progress in developing newer translational tasks by designing non-human animal versions of existing clinical tasks, or vice versa. Development of such analogous tasks will allow parallel studies to be conducted across different species using similar manipulations (e.g., pharmacological treatments to reverse deficits). Because the assessments described above involve laboratory-based operant behaviors, methods of data analysis can be identical (or at least very similar) across species. Moreover, advanced computational models that dissect separate subcomponents (e.g., reward sensitivity vs. learning rate) more precisely (102) or allow parametric modulations of brain function (103) may be applied similarly to data from different species. These important factors will mitigate the subjective interpretation of non-human animal and human behavioral responses to reward-related outcomes. Ultimately, the success of such cross-species behavioral tasks for drug discovery will hinge on the ability of the preclinical versions to accurately predict behavioral outcomes in the clinical versions, paving the way for the development of effective therapeutics for reward-related symptoms.