Modeling risky decision-making in nonhuman animals: shared core features

Abstract

Understanding the neural mechanisms of risky decision-making is critical to developing appropriate treatments for psychiatric disorders, problem gambling, and addiction to drugs of abuse. Probing neurobiological mechanisms requires the use of nonhuman animal models (particularly rhesus macaques, rats, and mice). However, there is considerable variation across species in risk preferences. Nevertheless, there are shared core features of risky decision-making present across species. As demonstrated with a wide variety of behavioral paradigms, modulators of risk preference observed in humans are readily replicated in model species. Thus, risky decision-making represents an important implementation of reward-guided decision-making that is feasibly modeled across species.

Introduction

Risk is a ubiquitous feature of our decision-making, whether we’re playing slot machines, determining when to merge into oncoming traffic, or testing out a new menu option at a favorite restaurant. For centuries, economists have known that humans are largely risk-averse for gains. For example, when offered a choice between a certain option of $40, and a risky option with 50% chance of paying out $20 and 50% chance of paying out $60, most people choose the safe $40. Although many studies have replicated such risk-aversion for gains in nonhuman animals, others have not. Indeed, cross-species differences in risk preferences are now quite well-documented [1–3]. That is, when nonhuman animals are offered risky versus safe nutritive rewards, their preferences range from risk-seeking to risk-averse. These preferences may be affected by species-specific foraging environments during evolution [4].

At first glance, the substantial inter-species differences in risk preferences might render human risky decision-making difficult to model in other species. This would be unfortunate, since nonhuman animals, particularly rats (Rattus norvegicus), rhesus macaques (Macaca mulatta), and mice (Mus musculus), are essential for probing the neurobiological mechanisms of behavior. They are also essential for developing effective treatments for conditions associated with maladaptive risky decision-making, such as problem gambling and addiction to drugs of abuse. However, for two distinct reasons, risky decision-making is both especially ideal and important to study in neurobiological model species:

1. The close relationship between psychiatric disorders and risky decision-making suggest that the latter is an important domain to study.

2. Risky decision-making is an excellent encapsulation of value-based decision-making generally, as it involves two parameters that can be independently manipulated: reward amount and win/loss probability. This allows neurobiologists to identify brain mechanisms of these different parameters, and their combination into subjective value.

3. Although baseline risk preferences can be quite different across species, shared core features (modulators of risk preference) suggest common neural and cognitive mechanisms.

Although risky decision-making is a classic problem in economics, psychology, and behavioral ecology, recent years have seen an explosion in the behavioral tasks used to study specific elements of risky decision-making in nonhuman animals [5]. Therefore, my goal is to review available behavioral evidence regarding risky decision-making in neurobiological model species--mainly macaques, rats, and mice.

What is risk?

The term risk has different meanings for behavioral economists, psychiatrists, and laypeople [6]. For a behavioral economist, risk is defined in terms of variance in possible outcomes. Thus, the gamble listed in the first paragraph (50% probability of winning $20, 50% probability of winning $60) is risky, even though there is no danger of an actual loss. A subject who chooses the gamble over the sure thing ($40) is said to be risk-seeking, as the two options have the same expected value (the sum of possible values multiplied by their probabilities of occurrence). Risk increases as variance increases, so an option with a 50% probability of winning $10 and a 50% probability of winning $70 is higher risk than the $20/$60 option references above. Psychiatrists and laypeople, by contrast, tend to frame risk in terms of harm. Thus, the former example would not be risky, because it involves no possible harm to the decider.

Here, I use the economic definition of risk, so harm or loss is not necessary for an option to be considered risky. I will also include those studies that involve outcomes of multiple types (in which it is impossible to equalize expected values across options, i.e., pellets and footshocks), so long as there is outcome variance.

Tasks used to model risky decision-making

The tasks used to study risky decision-making differ from species to species, in part because of divergent original purposes. Most of the studies of risky decision-making in rhesus monkeys were designed to prioritize recording from single neurons in awake, behaving animals [7]. In this case, constraints are high: behavior must be reliably replicable across sessions, and sessions must consist of hundreds or even thousands of trials each (although see [8]). Among these tasks, there are two broad categories: those in which probabilities and/or rewards are directly cued and can change from trial to trial [9–13], and those in which these parameters must be learned in blocks [8,14–16]. In both types, monkeys choose between two options associated with different probabilities and amounts of reward (typically juice). Thus, there is only one outcome type (reward), and expected values are easy to calculate and compare across options. Depending on the study, these choices may simply be between risky and safe options with always equalized expected values, or they may include low risk vs. high risk trials, or trials in which expected values are quite different.

There have been no risk studies in macaques that use punishment, only missed opportunity for reward. However, in Seo and Lee [17], monkeys face a strategic decision-making task similar to a gamble, and they are able to increment and decrement tokens that accrue toward an eventual payoff. Similar token tasks have been used in macaques for metacognitive studies with a risk element [18]. Future studies may use loss of tokens (instead of less reward, or zero reward) to more directly study risky decision-making.

Rodent risky decision-making tasks are much more varied, in part because they were originally developed not only to allow recording activity of single neurons, but also to perform pharmacological and lesion manipulations. Nevertheless, some of the tasks used in rats match the blocked macaque paradigm, in that choices are between a risky and a safe option with equivalent expected values (rats: [19,20]; mice: [21])

In the probability discounting task, rats choose between a small, certain reward and a large reward offered at different probabilities across blocks (increasing or decreasing within or across sessions) [22–24]. One disadvantage of this paradigm is the potential for perseveration: subjects initially learn that the lever that will eventually become risky has a reward probability of 100%, perhaps making it difficult to switch away from the risky option even as it becomes less rewarding. In the rodent betting task, rats are offered the choice between a fixed amount, and the chance to “bet” on a risky option that, half the time, gives double the fixed amount, and half the time gives nothing [25,26]. The fixed amounts change across blocks. Finally, the rodent slot machine task represents a departure from previous tasks used in rodents in that parameters change on a trial-to-trial basis [27,28]. Illuminated lights signal the probability of winning, and their number can change on each trial.

An additional set of tasks used in rats, but not typically in monkeys (although see [13]), involves outcomes that differ from one other on more than amount or probability of reward: possibility of shock or timeout, or a qualitatively distinct reward. For example, rats may choose between a small food pellet, and the option of a large food pellet with some probability of a footshock [29]. A subset of these attempt to model the Iowa Gambling Task (IGT), a popular paradigm in which humans repeatedly choose among four decks, each associated with different probabilities of winning and losing, and different reward parameters [30]. Over time, subjects learn to choose the advantageous deck: the one with the largest reward output over time. However, risk and long-term reward are (inversely) confounded in the options: those decks with higher variance also have lower expected values over the course of the experiment [6]. That is, the risky decks are also less optimal, which is not the case in paradigms in which expected values are equalized. Thus, the IGT cannot be used to determine a subject’s preference for risk or expected value uniquely [31]. Nevertheless, the parameters of the IGT may capture many of the decisions associated with problematic, impulsive behavior (drug-seeking, problem gambling, etc). The challenge here is how to model loss of reward in nonhuman animals. Solutions range from using quinine-laced food pellets (which are less rewarding than typical pellets [32,33]) to implementing timeouts [34–37]. While none of these quite matches the human paradigm, and require extensive training not used with humans, the behavioral responses seem to align with those observed in humans (see below).

What’s shared, what’s not

Importantly for studying neural mechanisms, humans, macaques, rats, and mice are all exquisitely sensitive to reward size and probability. Preferences closely track reward size and reward probability. Furthermore, in a subset of the tasks described above, it is possible to assess preference for risk itself. In these, expected values are equalized across options, but risk-seeking individuals choose the risky option more frequently, while risk-averse individuals choose the safe option more frequently.

Across many laboratories, rhesus macaques have been shown to be risk-seeking, meaning they prefer a risky option to a safe one of equivalent value [9–13] (although see [38]). Although this would seem to contrast with the risk aversion for gains pattern commonly observed in humans, placed in a similar situation, humans are less risk-averse than usual [39]. Thus, at least some of this species difference is due to unique aspects of the monkey paradigm (short intertrial intervals, small rewards, and so on: see [7]).

Furthermore, although baseline preferences may be slightly different between humans and macaques, thus far, modulators of risk preference are largely shared. Here, modulator refers to any factor that may influence risky choice aside from risk level itself, and may include environment, trial-to-trial effects, context, internal state, and learning patterns. Rhesus monkeys, like humans, prefer options in which probabilities are known (risky ones) to those in which probabilities are unknown (ambiguous ones) [9,40]. For example, most people prefer a 50% chance of winning $10 to a chance of winning $10 that is unknown, but ranges from 0–100% (note that the expected values of these options are equal). This effect, known as ambiguity aversion, has been replicated in rhesus monkeys by using colored bars to explicitly cue reward probabilities [9]. In addition, monkeys and humans both prefer risky options learned through experience to those explicitly cued (monkeys) or verbally described (humans) [41,42] (Figure 1), a phenomenon known as the description-experience gap. Macaques and humans alike overweight low probabilities (meaning they act as though rare events occur more often than they actually do) and underweight high probabilities (meaning they act as though very common events occur less often than they actually do) [12,43].

Figure 1.

The experience-description gap in risky choice, as assessed in rhesus monkeys (left, adapted from [41]) and humans (right, adapted from [42]). In both cases, there were two types of gambles: those explicitly cued (top row) and those whose probabilities were learned through experience (second row). There was also a safe option in each case (third row). Both monkeys and humans chose the risky option more often when it was learned through experience than when it was explicitly cued/described (bottom row).

Along with the larger number of behavioral tasks is a greater variety of risk preferences in rats, relative to macaques. When expected values are equalized, rats range from risk-seeking to risk-averse ([20,44], see [3] for an early review), with substantial individual differences within and between experiments. Unfortunately, the range of risk preferences in macaques is largely unknown, because the cost of each research subject makes testing large numbers prohibitive. Like macaques, however, rodents may share important modulatory effects on risky decision-making with humans, pointing to shared mechanism.

Shared among rodents, macaques, and humans is a tendency to overweight decisions on the basis of recent wins or losses: they all follow win-stay/lose-shift choice patterns. That is, having chosen the risky option and won, subjects are more likely to choose the risky option again than if they lost (rats: [45,46]; monkeys: [15]). Furthermore, in humans, risk-aversion is weaker when reward stakes are very small [47]; this also seems to be true in rats [26] and monkeys [48].

Rats and humans also show a near miss effect: when the stimulus is close to, but not quite, what is needed to obtain a reward, subjects treat the stimulus as if it were likely to yield a reward (Figure 2). Near-misses on slot machines may drive gambling in humans [49]. In rats, this has been demonstrated using the rodent slot machine task (described above). If, for example, five lights must be illuminated in order to win a reward, but four lights or fewer leads to no reward, rats treat the four light option as a near miss, and often approach to collect a reward [27,28].

Figure 2.

Near miss paradigms in rodents (left, adapted from [27]) and humans (right, adapted from [49]). Rodents face trials in which 0–2 lights illuminated is followed by no reward, but 3 lights illuminated is followed by a reward. 0–1 lights illuminated are treated as a clear loss, but 2 lights illuminated is treated as a near-miss. Humans face a slot-machine-like paradigm in which stimuli at the center must align in order to win. A near-miss is a misalignment by one row, while a clear loss is a misalignment by >1 row.

There are also broader factors that may determine risk preferences, some of which are shared across species. For example, in both rats [50] and humans (reviewed in [51]), males are more risk-seeking than females. However, this pattern is reversed in the IGT [52,53], and males choose the less risky, advantageous option more frequently than females.

Finally, the extensive studies on rodent forms of the IGT have allowed detailed comparisons with human behavior on this paradigm. Like humans, mice and rats gradually learn to choose the option that is most rewarding in the long term. Their learning curves even look similar [32]. Across species, there are intriguing individual differences as well, such that some authors posit that problem gambling may be modeled in a subset of rodents [53].

Why risky decision-making in particular?

Risky decision-making has intuitive links with a number of psychiatric disorders, as well as impulse control issues in healthy subjects. Risky decision-making in the laboratory has obvious links with problem gambling. However, these decisions are abnormal across many other disorders as well: drug abusers are more risk-seeking, even when currently abstinent [54,55]; patients with major depressive disorder may show enhanced risk aversion (on the IGT) [56]; dispositional anxiety is associated with enhanced risk aversion [57]. Thus, impairments in risky decision-making may represent a trait underlying many psychiatric disorders.

Furthermore, there is some evidence that modulatory effects (that are shared across species, described above) are important for understanding the nature of the abnormality in risky-decision making in specific disorders. For example, drug abusers may be more risk-seeking than healthy controls because they fail to shift their choices following a loss on the risky option [55]. Because these modulatory effects can be effectively modeled in nonhuman animals, it is possible both to investigate underlying neural mechanisms at a much finer-grained scale than is feasible in humans and to develop and test pharmacological treatments.

Conclusions

Risky decision-making represents an important and tractable problem for neurobiologists. Luckily, the wealth of well-designed behavioral paradigms allows us to probe specific aspects of these decisions across species. In doing so, shared features have emerged: modulatory effects (such as a dependence on how options are learned or presented) originally observed in humans can be readily replicated in nonhuman animals. Finally, because the circuitry underlying risky decision-making—namely, orbitofrontal and anterior cingulate cortical-basal ganglia loops [58]—is also largely conserved across rodents, nonhuman primates, and humans [59], it is possible to dissect the neurobiology of these behaviors in model species.

Highlights.

• Animal models of risky decision-making are important for probing neurobiology.

• Tasks have been developed to evaluate risky decision-making in rodents and macaques.

• Modulatory effects (context, environment, etc) in are preserved across species.