Advances in understanding meso-cortico-limbic-striatal systems mediating risky reward seeking

Abstract

The risk of an aversive consequence occurring as the result of a reward-seeking action can have a profound effect on subsequent behavior. Such aversive events can be described as punishers, as they decrease the probability that the same action will be produced again in the future and increase the exploration of less risky alternatives. Punishment can involve the omission of an expected rewarding event (‘negative’ punishment) or the addition of an unpleasant event (‘positive’ punishment). Although many individuals adaptively navigate situations associated with the risk of negative or positive punishment, those suffering from substance use disorders or behavioral addictions tend to be less able to curtail addictive behaviors despite the aversive consequences associated with them. Here, we discuss the psychological processes underpinning reward seeking despite the risk of negative and positive punishment and consider how behavioral assays in animals have been employed to provide insights into the neural mechanisms underlying addictive disorders. We then review the critical contributions of dopamine signaling to punishment learning and risky reward seeking, and address the roles of interconnected ventral striatal, cortical, and amygdala regions to these processes. We conclude by discussing the ample opportunities for future study to clarify critical gaps in the literature, particularly as related to delineating neural contributions to distinct phases of the risky decision-making process.

Graphical Abstract

Reward seeking can be potently modulated by the risk of an aversive event occurring: threat of reward omission (negative punishment) or concomitant delivery of an unpleasant stimulus (positive punishment) both cause animals to alter their behavior. In this review, we provide an overview of negative and positive punishment-based models of risky reward seeking, discussing the relevance of these approaches to understanding neuropsychiatric diseases such as substance use and alcohol use disorders. We then review literature establishing the necessity of nodes within a meso-cortico-limbic-striatal network that regulate these two types of risky reward seeking.

Legend: Pr (Probability), BLA (basolateral amygdala), D1R (dopamine D1-receptor), D2R (dopamine D2-receptor), IL (infralimbic cortex), LHb (lateral habenula), lOFC (lateral orbitofrontal cortex), mOFC (medial orbitofrontal cortex), mPFC (medial prefrontal cortex), NAcCore (nucleus accumbens core), NAcShell (nucleus accumbens shell), PAG (periaqueductal gray), PL (prelimbic cortex), RMTg (rostromedial tegmental nucleus), VTA (ventral tegmental area).

Introduction

Nearly all choices carry some degree of risk, defined as the possibility that an undesirable outcome might arise from a choice and subsequent course of action (Yates and Stone, 1992). Typically, the risk of an aversive event occurring following a particular action biases future behavior towards other, safer actions. However, this adaptive flexibility is compromised in a variety of neuropsychiatric conditions. Of particular note, alcohol use disorders (AUDs), substance use disorders (SUDs), and behavioral addictions are characterized by continued engagement in behaviors associated with the risk of both short- and long-term adverse consequences (Everitt et al., 2008; Feil et al., 2010; Figee et al., 2016).

Laboratory studies in humans and non-human animals have empirically assessed risk-taking by juxtaposing probabilistically delivered aversive events with reinforcement-seeking behavior or reward delivery (Bechara et al., 1994; Ersche et al., 2016; Jean-Richard-Dit-Bressel et al., 2018; Levy, 2017; Orsini et al., 2015). This general approach is designed to engender a state of internal conflict whereby actions associated with reward are tempered by the motivation to avoid unwanted outcomes (Miller, 1944). An organism’s response to a risky situation is dependent on multiple factors, including its subjective or interoceptive state, the probability of reward or punishment delivery, and the relative strength of these outcomes (and the salience of the stimuli that signal them) (Cardinal and Howes, 2005; Church, 1963; Simon et al., 2009). Studies examining these processes have begun to reveal an interconnected network of brain regions that regulate aspects of risky decision-making, with many of these same neural systems functioning aberrantly in SUDs and animal models of addiction (Everitt et al., 2008; Halladay et al., 2020; Marchant et al., 2013; Pascoli et al., 2015; Radke et al., 2015; Vanderschuren et al., 2017).

In this review, we first discuss approaches commonly employed to study risky reward seeking, focusing mostly on tasks that employ forms of punishment to alter behavior, and describe how these assays have been adapted to study addiction-relevant behavior (Fig. 1). We then review data from studies investigating the neural regulation of risky behaviors in laboratory animals and to a lesser extent, humans. Throughout, we detail pertinent data addressing the contributions of interconnected meso-cortico-limbic-striatal structures to risky decision-making (Fig. 2). This focus on structures interconnected with the midbrain dopamine (DA) system arises from the well-established link between DA dysfunction and psychiatric illnesses characterized by maladaptive risk-seeking behavior (e.g., SUDs) (Feil et al., 2010; Lubman et al., 2004; Volkow and Morales, 2015). We refer the reader to excellent recent reviews that address relevant regions and transmitter systems not discussed here (Jean-Richard-Dit-Bressel et al., 2018; Orsini et al., 2015).

Figure 1.

Cartoon depictions of common methods for assessing risky reward seeking in rodents. (A) During performance of a negatively punished ‘probabilistic risk discounting’ task, animals learn that one lever is associated with a large amount of reward (depicted as a large liquid droplet) but is also associated with a particular probability (Pr) of reward omission (depicted as a droplet + ‘No’ symbol), while another lever guarantees delivery of a small amount of reward (depicted as a small liquid droplet). If the animal presses the large reward lever (green downward arrow) and reward is omitted, animals are more likely to switch away from the large reward option, and towards the small, certain option on the next trial (bottom panel). (B) As in A, an animal trained on a ‘risky decision-making’ task initially discriminates between two levers, one that delivers a large volume of liquid reward, and one that delivers a small volume. Selection of the large reward option is probabilistically (Pr) associated with footshock (yellow lightning bolts) punishment (top), which causes animals to flexibly shift their actions towards the guaranteed, safe option (bottom). (C) Positive punishment-based approaches have been used to examine the mechanisms contributing to ‘compulsive’ drug-related behavior. In this example, an animal self-administering cocaine (syringe) is subjected to probabilistically delivered footshocks concomitant with cocaine-seeking lever-presses (top). In animals sensitive to punishment, experience with footshock can cause animals to become more reticent, engaging in fewer bouts of drug-seeking (bottom). Pr (probability).

Figure 2.

Neural pathways implicated in risky reward-seeking behaviors. Regions, pathways, and receptors are outlined in different colors to depict their involvement in each form of ‘risk’: dark blue represents circuit elements that have been shown to be involved in positive punishment-based assays, green represents circuit elements involved in negative punishment-based assays, and light blue represents circuit elements that have been linked to both behaviors. Note, anatomical connections between regions that have not explicitly been implicated in risky reward seeking are not displayed. For simplicity, projections from VTA have been mostly eliminated. BLA (basolateral amygdala), D1R (dopamine D1-receptor), D2R (dopamine D2-receptor), IL (infralimbic cortex), LHb (lateral habenula), lOFC (lateral orbitofrontal cortex), mOFC (medial orbitofrontal cortex), mPFC (medial prefrontal cortex), NAcCore (nucleus accumbens core), NAcShell (nucleus accumbens shell), PAG (periaqueductal gray), PL (prelimbic cortex), RMTg (rostromedial tegmental nucleus), VTA (ventral tegmental area).

Assaying reward seeking in the face of risk

When operationalized by behavioral scientists, ‘risk’ is often introduced by associating some proportion of reward-seeking actions with punishment. Punishment is the response-contingent occurrence of an aversive event that causes a subsequent decrement in the behavior perceived to have caused the punishment (for review, see Jean-Richard-Dit-Bressel et al., 2018). Punishments can be thought as being either ‘positive’ or ‘negative,’ in the sense that positive punishment directly results in the addition of an unpleasant outcome (i.e., delivery of a stimulus that causes discomfort or pain), while negative punishment involves the omission of a reinforcing outcome (i.e., risk of loss and reinforcer uncertainty). These two forms of punishment reflect fundamental instrumental learning mechanisms that are apparent in everyday life. Speeding to avoid being late for work comes with the risk of positive punishment - a traffic citation or accident. The possibility of negative punishment also pervades decisions both small (i.e., buying a losing lottery ticket) and large (i.e., purchasing property in a ‘seller’s market’) where there is a risk of making the wrong choice and losing out. Indeed, influential neuroeconomic theories have been built around the power of punishment, particularly the fear of loss, to shape human behavior (Kahneman and Tversky, 1979; Loewenstein et al., 2007).

In laboratory animals (primarily rodents and non-human primates), reward seeking despite loss-related (negative) punishment can be assayed using probabilistic risk discounting or multi-armed bandit tasks (Cardinal and Howes, 2005; St. Onge and Floresco, 2009; Sutton and Barto, 1998). These tasks present subjects with a choice between multiple reward options that are each associated with distinct probabilities of reward omission. In the case of probabilistic risk discounting, rats can be trained to discriminate between two levers, one that delivers a guaranteed small amount of reward, and another that is associated with a large volume of reward, but also a certain probability of reward omission (Fig. 1A, top). Unexpected reward omission following selection of the large ‘risky’ option (i.e., a ‘loss’) normally decreases the probability of this action being performed on subsequent trials, while increasing the exploration of other opportunities to obtain reinforcement (Fig. 1A, bottom). On probabilistic discounting tasks, reward uncertainty increases over discrete blocks of choice trials such that most animals shift their behavior away from preferred ‘risky’ reward option as losses accumulate (Cardinal and Howes, 2005).

Behavioral performance on negative punishment tasks can be statistically modeled using reinforcement learning approaches, which can reveal latent variables, such as value, influencing performance (Constantinople et al., 2019b; Neftci and Averbeck, 2019). Microstructural, trial-by-trial analysis of behavior can also be employed to identify performance strategies adopted by the subject, including those associated with reward sensitivity (e.g., ‘win-stay’– selecting a risky option after ‘winning’ on the previous trial) or negative-feedback sensitivity (e.g., ‘lose-shift’ – switching to the safe or low risk option after ‘losing’ on the previous trial) (Estes, 1994; Orsini et al., 2015). Such computational analyses have provided insight into the psychological and biological processes that mediate negative punishment-based reward seeking.

‘Positive’ punishment tests can be thought of as ‘conflict’ tasks (as they were often originally described; Geller and Seifter, 1960; Vogel et al., 1971) due to the motivational competition produced by the desire to avoid noxious stimuli on one hand, and the prepotent drive to perform an instrumental reward-seeking action on the other. In a typical experimental preparation, a rodent (commonly in a state of food or water deprivation) can be asked to lever-press for reinforcement despite the possibility of footshock delivery concomitant with some proportion of presses (Estes, 1944; Geller and Seifter, 1960). Other noxious stimuli, such as pressurized air puffs (Spealman, 1978) or adulteration of liquid reinforcement with a bitter tastant (Wolffgramm, 1991; Wolffgramm and Heyne, 1991), have also be used as punishers in this kind of task.

The risk of positive punishment can profoundly affect behavior: reward seeking might become more infrequent, anxiety-related changes in posture and approach behavior can emerge, and the animal might begin to investigate other sources of reinforcement, if available (Geller et al., 1962; Geller and Seifter, 1960; Grant and Mackintosh, 1963; Halladay et al., 2020; Hunt and Brady, 1955; Millan and Brocco, 2003; Resstel et al., 2008; Van Der Poel, 1979; Vogel et al., 1971). Positive punishment-based decision-making has been assessed in rats (Simon et al., 2009) and mice (Glover et al., 2020) using a risky decision-making task, whereby animals learn to discriminate between a response option that delivers a guaranteed small reward, and another option delivers a large reward plus a probabilistically determined footshock (Fig. 1B, top). Like reward omission on the probabilistic risk discounting task, footshock punishment generally causes animals to shift their choice preference away from the preferred large reward and towards the explicitly safe small reward option over time (Fig. 1B, bottom).

A variety of experimenter-controlled factors contribute to the magnitude of effect that positive punishment can have on reward-related behaviors. To maintain a level of reinforcement seeking that allows for the detection of experimentally-induced increases in punishment sensitivity, punishment is usually delivered probabilistically (non-continuously), e.g., after a predetermined fraction of instrumental actions (Estes, 1944). In the case of discrete, temporally specific punishers like footshock, stimulus intensity can be titrated such that reinforcement-seeking behavior is not overly counteracted by opposing defensive reactions such as freezing (Estes, 1944; Hunt and Brady, 1955). Experiments must also be carefully designed to accommodate factors that vary due to individual differences. For example, positive punishment in the form of footshock has been shown to have greater inhibitory effects on food reward seeking in female, as compared to male, rodents (Chowdhury et al., 2019; Jacobs and Moghaddam, 2020; Orsini et al., 2016). This effect may not be due solely to sex-related differences in reactivity to the unconditioned stimulus (e.g., footshock) (Archer, 1975; Orsini et al., 2016), suggesting that the reported changes in punishment sensitivity reflect a cognitive phenomenon. Interestingly, despite a greater tendency to avoid risk in drug-free contexts, females are more likely than males to endure positive punishment when responding for drug rewards, including alcohol (Fulenwider et al., 2019; Radke et al., 2019a; Sneddon et al., 2020, but see Sneddon et al., 2019) and opioids (Monroe and Radke, 2020).

Susceptibility to the effect of positive punishment on reward seeking also appears to exist on a continuum, with some individuals being relatively more resistant to punishment-induced inhibition than others (Giuliano et al., 2018; Marchant et al., 2018; Spoelder et al., 2017, 2015). This spectrum of punishment sensitivity may be partially explained by genetic factors: for example, laboratory rats bred to prefer alcohol are often relatively resistant to positive punishment of alcohol drinking (in comparison to their non-preferring counterparts) (Giuliano et al., 2018; Houck et al., 2019; Timme et al., 2020), while various strains of commonly used laboratory mice exhibit marked differences in susceptibility to footshock or quinine-induced suppression of alcohol seeking (Halladay et al., 2017). Finally, it is important to recognize that differences in resistance to punishment can arise from differences in sensitivity to the reinforces or punishments used in the study, and thus controls are necessary to ensure that experimental manipulations do not alter factors such as reactivity to footshock or taste sensitivity.

Punishment-based models of risk seeking relevant to addictions

Abnormalities in risk processing are thought to contribute to the etiology of addictions (Crawford et al., 2003; Hawkins et al., 1992; Pilgrim et al., 2006) and the intractable patterns of compulsive reward seeking that characterize them (Feil et al., 2010; Figee et al., 2016; Koob and Volkow, 2010). One hypothesis is that AUDs, SUDs, and behavioral addictions arise from deficits in the ability to adjust behavior after feedback from ‘positive’ (e.g., adverse health) and ‘negative’ (e.g., monetary losses) punishment experience (Grant et al., 2010), and that these deficits may be triggered or exacerbated by chronic drug exposure. Performance on negative punishment-based tests of risky decision making such as the Iowa Gambling Task (IGT) are reliably impaired in SUD patients, with these individuals preferring options associated with large rewards despite the risk of large losses (Bechara et al., 2001; Bolla et al., 2005; Fishbein et al., 2005; Grant et al., 2000). Impairments in the ability of positive punishment to affect behavior have also been reported in cocaine-addicted individuals (Ersche et al., 2016). These findings have led to the development of animal models assessing the impact of self-administration of, or passive exposure to, abused substances on reward seeking despite punishment.

Consistent with a conceptual framework in which risk processing is impaired in addiction, a number of preclinical studies have shown that cocaine or alcohol exposure reduces sensitivity to losses in the context of negative punishment. For example, the preference for risky options on a rat gambling task was shown to be enhanced in a subset of animals by chronic cocaine self-administration (Cocker et al., 2020; Ferland and Winstanley, 2017), and relapse to cocaine use following abstinence was reported to be stronger in those rats that exhibited greater risk preference (Cocker et al., 2020). These observations are in line with human addiction studies showing that impaired decision-making is predictive of poor treatment outcomes (Darke et al., 2012; Stevens et al., 2013). In another clinical parallel, alcohol exposure during adolescence, which is a known risk factor for the development of AUDs (for review, see Crews et al., 2016), has also been shown to produce a long-lasting enhancement of risk preference as assessed by a loss-related (probabilistic discounting) negative punishment task in rats (Boutros et al., 2015; McMurray et al., 2016; Nasrallah et al., 2009; Schindler et al., 2016).

The ability of positive punishment to curtail drug-seeking behavior in preclinical models can also be altered by drug exposure (for review, see Hopf and Lesscher, 2014; Vanderschuren et al., 2017). Positive punishment-insensitive drug intake was first reported in the literature by Wolffgramm and colleagues (1991) who demonstrated that rats given nine months of alcohol access continued to drink more despite the adulteration of drinking liquid by the bitter tastant quinine, which had a profound punishing effect in alcohol-naïve control rats. Complementary data have also been provided by experiments in rodents whereby drug-seeking or -taking actions are paired with a more temporally restricted punishing stimulus like mild footshock (Fig. 1C). Early studies in monkeys demonstrated that electric shock punishment could inhibit stimulant self-administration, with parameters including stimulant dose and shock intensity modulating the magnitude of behavioral inhibition (Bergman and Johanson, 1981; Grove and Schuster, 1974; Johanson, 1977). More recent investigations in rodents have extended these findings, showing that long-term access to drugs of abuse results in punishment insensitivity in at least some animals (Deroche-Gamonet, 2004; Vanderschuren and Everitt, 2004), paralleling the clinical observation that SUDs develop in only a minority of individuals that partake in drug use (Anthony et al., 1994).

Punishment insensitivity following extended access or exposure to addictive drugs has been demonstrated for cocaine (Chen et al., 2013; Deroche-Gamonet, 2004; Kasanetz et al., 2010; Mitchell et al., 2014; Pelloux et al., 2007; Vanderschuren and Everitt, 2004) and alcohol (Barbier et al., 2017; Hopf et al., 2010; Houck et al., 2019; Kimbrough et al., 2017; Radke et al., 2017; Vendruscolo et al., 2012), as well as other addictive substances such as methamphetamine (Cadet et al., 2016; Torres et al., 2017) and fentanyl (Monroe and Radke, 2020). Extended cocaine access also causes cues associated with footshock to lose their ability to inhibit drug seeking, which is not true of extended exposure to natural rewards like sucrose (Vanderschuren and Everitt, 2004). Notably, the developmental trajectory of this compulsive-like phenotype often aligns with the progressive exacerbation of compulsive drug seeking that occurs in individuals with SUDs (for review, see Everitt and Robbins, 2005).

Positive punishment-based models have generated insights into the component psychological processes contributing to compulsive SUDs. For example, stress experience, which is a known etiological and precipitating factor in SUDs (for review, see Koob, 2009), has been shown to interact with factors such as genetics and sex to produce distinct trajectories of positive punishment-induced changes in reward seeking (Edwards et al., 2013; Radke et al., 2019a; Shaw et al., 2020). Chronic stress in mice (Edwards et al., 2013; Shaw et al., 2020) or early life adversity in rats (Radke et al., 2019a) each diminish the impact of quinine-based punishment on alcohol seeking, consistent with the idea that stressors increase compulsive drug-related behaviors.

Another parameter that can affect punisher efficacy in these models is whether punishment is applied to appetitive (drug-seeking) versus consummatory (drug-taking) actions (Everitt et al., 2018; Lüscher et al., 2020). It has been suggested that these two drug-related behaviors are mediated by separate psychological processes, and thus may be differentially susceptible to factors that contribute to the development of compulsivity, such as punishment sensitivity or habit formation (Everitt and Robbins, 2005; Lüscher et al., 2020). For example, Pelloux and colleagues (2015) behaviorally dissociated an instrumental cocaine-seeking action (e.g., press one lever for access to cocaine) from a separate, cocaine-taking action that became available after rats made a requisite number of seeking responses (e.g., press a second lever to receive cocaine infusion). They found that footshock punishment paired with the seeking response inhibited continued cocaine seeking, while the same punishment paired with the cocaine-taking action was less effective. These data further highlight how a myriad of variables moderate the effects of punishment on reward-seeking behavior, and imply there may be similarly complex factors, varying across individuals and situations, influencing how punishment affects compulsive drug-taking in AUDs and SUDs.

Armed with assays and models of punishment, researchers have worked to illuminate the neural underpinnings of risky decision-making (Fig. 2) – most often studied in drug-naïve animals working for natural rewards such as food – and the dysfunctions that arise in these neural systems following drug exposure. Studies employing the risk of loss (negative punishment) to affect behavior have provided valuable insights into the neural mechanisms that underlie reinforcement learning (Cohen et al., 2012; Fiorillo et al., 2013; Keiflin and Janak, 2015; Schultz, 2013; Schultz et al., 1997) as well as the circuits and neuromodulators that contribute to cost/benefit decision-making (Bercovici et al., 2018; Floresco et al., 2018; Floresco and Whelan, 2009; Jenni et al., 2017; Piantadosi et al., 2016; St. Onge et al., 2012b; Stopper and Floresco, 2011, 2014; Sugam et al., 2012; Zeeb et al., 2009, 2015). Many of the mechanisms uncovered thus far are shared with those contributing to reward seeking under risk of positive punishment, which encompass a network of meso-cortico-limbic-striatal regions and associated neuromodulatory systems critical to aversion and behavioral flexibility (Jean-Richard Dit Bressel and McNally, 2014; Jean-Richard-dit-Bressel and McNally, 2016; Jean-Richard-Dit-Bressel and McNally, 2015; Kim et al., 2011; Pascoli et al., 2015; Piantadosi et al., 2017; Simon et al., 2009; Vento et al., 2017).

Midbrain dopamine: encoding risk prediction, uncertainty, and punishment

Central among the known neural substrates underlying risky reward seeking are mesolimbic-projecting DA neurons of the midbrain ventral tegmental area (VTA). Alterations in DA function have been suggested to contribute to the abnormal risk-based neurocomputations evident in a variety of neuropsychiatric illnesses and a raft of studies have reported DA system dysregulation in AUDs and SUDs (for review, see Trifilieff and Martinez, 2014; Volkow et al., 2009). Behavioral addictions have also been linked to DA dysfunction: patients with restless-leg syndrome or Parkinson’s disease who receive treatment with DA-receptor agonists occasionally develop iatrogenic gambling addiction, suggesting that excessive risk-taking can be produced by DA receptor activation (Grall-Bronnec et al., 2018; Moore et al., 2014).

Dopamine, prediction error and risk

Based on the clinical evidence linking addiction with DA dysfunction and the fact that laboratory studies show that most drugs of abuse affect the function of mesolimbic-projecting DA neurons (Di Chiara and Imperato, 1988; Sulzer, 2011; Wise and Bozarth, 1998; Wise and Robble, 2020), work in rodents and non-human primates has attempted to identify the mechanisms (both psychological and cellular) by which the DA system, and perturbations of DA function, can affect risky reward seeking. This work has often been based around the observation that phasic signaling of mesolimbic DA neurons links cues or actions with rewards (Saddoris et al., 2015; Saunders et al., 2018) and encodes reward prediction error (RPE) – a discrepancy between an expected and received outcome that can be positive (outcome better than expected) or, as in the case of punishment, negative (outcome worse than expected) (Cohen et al., 2012; Fiorillo et al., 2013; Keiflin and Janak, 2015; Schultz, 2013; Schultz et al., 1997).

Reward learning models predict that these RPE signals are key to informing future actions or decisions by updating expectations based on knowledge accrued from prior experiences (Pearce and Hall, 1980; Rescorla and Wagner, 1972). Direct experimental evidence for this supposition comes from studies in rodents that have artificially instantiated RPEs via optogenetic manipulation of VTA DA neurons, such that stimulation of these neurons mimics positive RPEs and produces cue-related learning, while their inhibition can simulate negative RPEs and improve learning related to reward omissions (Chang et al., 2016; Steinberg et al., 2013). Of particular relevance to the current review, electrical stimulation of VTA neurons delivered during unexpected reward omissions has been shown to increase risky choices in rats performing a risk-discounting task, potentially by counteracting the ‘loss signal’ that would be expected to occur in VTA DA neurons as a result of negative RPE encoding (Stopper et al., 2014b). Thus, DA coding of RPEs appears central to the ability of feedback to adapt reward seeking in the face of negative punishment risk.

DA neurons can also contribute to risky reward seeking through certain non-RPE mechanisms. For example, DA neurons have been suggested to encode the degree of reward uncertainty associated with an action (Schultz, 2007). This takes the form of a ramping in neuronal activity prior to reward delivery that scales with the probability of reward omission, with maximal activity evident on the most uncertain trials (Fiorillo, 2003). This uncertainty-induced ramping of DA activity could reinforce risky actions, a computational process which may be maladaptively co-opted in disorders like addiction which are characterized by aberrant DA function (Di Chiara, 1999; Keiflin and Janak, 2015; Redish, 2004). Indeed, optogenetic self-stimulation of VTA DA neurons in mice produces compulsive reinforcement seeking that is insensitive to footshock punishment, further implying that DA hyperactivity is sufficient to produce excessively risky behavior (Pascoli et al., 2015).

Finally, the DA system may mediate aspects of risky reward seeking through the direct signaling of aversive events. While classic studies in monkeys and rats showed preferential DA firing to expectation violations for rewarding events (Mirenowicz and Schultz, 1996; Ungless et al., 2004), there is now compelling evidence from multiple species that subpopulations of DA neurons in the VTA and substantia nigra can be excited by aversive stimuli and cues that predict them (Badrinarayan et al., 2012; de Jong et al., 2019; Lammel et al., 2011, 2012; Matsumoto and Hikosaka, 2009a; McCutcheon et al., 2012). These aversion-encoding DA neurons are largely distinct from RPE-encoding DA neurons, as they tend to be localized more medially within the VTA (or laterally in the substantia nigra) and also express distinct molecular machinery (e.g., vesicular glutamate transporter 2) (Lerner et al., 2015; Verharen et al., 2020b). By encoding aversive or punishing outcomes, the activity of this subset of DA neurons could provide an important neural signal to multiple processes relevant to risk assessment (Oleson et al., 2012; Park and Moghaddam, 2017; Pultorak et al., 2018; Stelly et al., 2019; Verharen et al., 2020a; Wenzel et al., 2018).

Upstream DA modulators

DA neurons are positioned to affect risky reward seeking through multiple processes, from revising expectations and encoding uncertainty about specific courses of action, to signaling the aversive consequences of these actions. These functions are integrated across a wider network by brain regions upstream and downstream of midbrain DA neurons (Fig. 2). Particularly noteworthy in the context of risk is an upstream trisynaptic pathway connecting the glutamatergic lateral habenula (LHb) to the primarily gamma-aminobutyric acid (GABA)-expressing rostromedial tegmental nucleus (RMTg) and on to VTA DA neurons (Bromberg-Martin and Hikosaka, 2011; Hong et al., 2011; Matsumoto and Hikosaka, 2009b, 2007). A number of recent studies in rodents and non-human primates have suggested that aversive events activate the LHb to RMTg pathway, resulting in phasic decreases in DA neuron activation consistent with negative RPEs (Hong et al., 2011; Jhou et al., 2009; Li et al., 2019). In rats, pharmacological inactivation of the LHb or the RMTg decreased risk seeking in a probabilistically reinforced decision making context, suggesting that each structure is necessary for optimal decision-making in the face of loss-related feedback (Stopper and Floresco, 2014).

Direct evidence that phasic activation of these nuclei contribute to action selection in the face of negative punishment risk comes from a separate study utilizing the same behavioral assay, showing that pairing electrical stimulation of the LHb or RMTg with the delivery of risky wins (i.e., wins occurring despite a low expectation of reward, potentially causing positive RPEs in DA neurons) decreases risky actions, whereas stimulation of the LHb during the pre-choice deliberation period elevates response latencies and biases actions away from risky options (Stopper and Floresco, 2014). These findings suggest that outcome-related activity in the LHb may instantiate negative RPEs in DA neurons that are necessary for the adjustment of behavior following negative punishment.

Emerging evidence suggests that positively punished reward seeking engages communication between many of these same structures that ultimately converge on VTA DA neurons (Jhou and Vento, 2019). Initial studies in rats and monkeys suggested that the LHb contributes to the vigor of reward seeking, but was not directly involved in positive punishment-driven shifts in reward seeking (Jean-Richard Dit Bressel and McNally, 2014) or the ability of aversive conditioned stimuli and outcomes to affect DAergic RPE signals (Tian and Uchida, 2015). However, more recent findings in rats indicate that inputs from the LHb to the RMTg contribute to punished suppression of reward seeking when the punishment is maximally unpredictable (Li et al., 2019). The RMTg may be supported in performing these more nuanced computations by other regions that relay distinct types of information during punished decision-making. For example, optogenetically silencing projections from the prelimbic region of the medial prefrontal cortex and the parabrachial nucleus to the RMTg increases punishment resistance in rats when silencing occurs during choice deliberation and outcome evaluation, respectively (Li et al., 2019). In turn, optogenetically silencing RMTg terminals in the VTA has been shown to increase risky reward seeking when laser delivery was timed to occur either when rats received footshock punishment or when animals were deciding to engage in reward seeking that might incur punishment (Vento et al., 2017).

Other inhibitory afferents to the VTA can also regulate the ability of positive punishment to inhibit reward seeking. In mice, optogenetic excitation of an inhibitory input from lateral hypothalamus to the VTA, for example, has been reported to decrease compulsive-like sucrose seeking despite the threat of footshock (Nieh et al., 2015), while hyperexcitability of an orbitofrontal cortex excitatory input to the VTA can drive compulsive actions (Pascoli et al., 2015). Of note, unlike instrumental punishment, the ability of cues that predict potential punishment to suppress sucrose seeking may not depend on VTA DA neurons in some scenarios, as a recent investigation showed that chemogenetic activation of these neurons in rats does not affect cue-induced behavioral inhibition (Verharen et al., 2020a). Although there remain many unknowns regarding the specific contributions of VTA to risky reward seeking, these findings broadly support the idea that the regulation of VTA neuron excitability by multiple upstream structures is critical to punishment risk-induced changes in reward seeking.

Nucleus accumbens: utilizing dopamine to adjust risky behavior

Once inputs converge and are computed at the level of DA neurons, these neurons send signals to multiple projection targets to affect risky decision-making. Of the forebrain structures targeted by VTA neurons, mesolimbic projections to the nucleus accumbens (NAc) have long been recognized as integral to reward processing, learning, and incentive motivation (Berridge, 2012; de Jong et al., 2019; Radke et al., 2019b; Robbins and Everitt, 1992; Stern and Passingham, 1996; Sugam et al., 2012). In rodents, the activity of NAc neurons and DA release within this nucleus encodes outcomes and drives reward-seeking actions (Ambroggi et al., 2008, 2011; Ikemoto and Panksepp, 1999; Nicola, 2010; Saunders et al., 2018; Sugam et al., 2012, 2014). In a more explicit link to risky decision-making, phasic DA release in the NAc is correlated with reward preference and scales with outcomes in a manner consistent with RPE encoding in rats performing a loss-based punishment task (Sugam et al., 2012). DA levels in the NAc of rats also track parameters relevant to motivational state during performance on a negative-punishment based probabilistic discounting task, including reward rate and subjective preference (St. Onge et al., 2012a). Hence, a plausible hypothesis is that DA release in the NAc reflects changes in motivation related to the presence of risky or volatile patterns of reinforcement.

In line with this view, elevated stimulus-evoked DA signaling within the NAc is associated with increased (loss-based) risk preference and enhanced incentive motivation in adult rats that were exposed to alcohol during adolescence (Nasrallah et al., 2011; Spoelder et al., 2015). Furthermore, these behavioral abnormalities can be reversed by the normalization of VTA DA neuron activity produced by systemic administration of a GABA agonist, implying a causal contribution of this region to risk-based computations (Schindler et al., 2016). Pharmacological and optogenetic manipulations have directly demonstrated the necessity of the NAc and its DA innervation to updating actions based on changes in risk probability (Cocker et al., 2012; Floresco et al., 2018; Freels et al., 2020; Mai et al., 2015; St. Onge et al., 2012a; Stopper et al., 2013; Stopper and Floresco, 2011; Sugam et al., 2012; Zalocusky et al., 2016). For instance, inhibiting neural activity within the NAc of rats decreases risk-seeking behavior in loss-related punishment tasks (Cardinal and Howes, 2005; Stopper and Floresco, 2011). Dopamine itself can alter neural activity and plasticity within the NAc by affecting its cognate D1-receptor (D1R) and D2-receptor (D2R), receptors which are expressed in mostly non-overlapping populations medium spiny projections neurons in the striatum (Baimel et al., 2019; Gerfen et al., 1990; Scudder et al., 2018) and which induce opposite effects on cellular excitability, with D1R activation increasing excitability and D2R activation decreasing excitability (for review, see Tritsch and Sabatini, 2012).

Studies examining the role of these NAc DA receptors in risk-related actions have generally found that the D1R family of DA receptors influence loss-related decisions, while the D2R family can mediate negative feedback more generally across punishment types. In rats, intra-NAc microinfusion of the D1R/D2R antagonist flupenthixol or the D1R-specific antagonist SCH23390 decreases risk seeking, reducing the preference for reward options associated with losses (Mai et al., 2015; Stopper et al., 2013). On the other hand, pharmacological blockade (via eticlopride) or activation (via quinpirole) of NAc D2Rs was found to be without effect when tested on the same negative punishment-based task (Stopper et al., 2013). Interestingly, activation of D2Rs on NAc neurons in rats can promote risky behavior in some instances, possibly by preventing dynamic changes in D2R activation that signal unexpected reward omission and promote shifts in choice following negative feedback (Zalocusky et al., 2016). Indeed, measuring changes in calcium activity in D2R-expressing NAc neurons revealed that these neurons are active following losses, and their activity during subsequent pre-choice task phases biases rats towards less preferred, but safe (guaranteed) reward options (Zalocusky et al., 2016). In keeping with the idea that the activation of D2R-expressing NAc neurons opposes risk-seeking, this same study demonstrated that optogenetic activation of these neurons induces a bias away from risky actions, specifically in a subset of rats identified as risk-preferring.

Although less numerous, data from positive-punishment based tasks also generally support the idea that D2R-expressing NAc neurons signal negative feedback related to aversive events. For example, infusion of the D2R/D3R agonist quinpirole into the NAc of rats reduces risk-taking, and NAc D2R gene expression is inversely correlated with adolescent risk-taking in the same footshock-based punishment task (Mitchell et al., 2014). Blockade of NAc D2Rs with sulpiride on a conflict-based reward-seeking task resulted in rats expressing less avoidance of a cue associated with prior punishment (while D1R antagonism had the opposite effect) (Nguyen et al., 2018). These findings support prior observations that D2Rs in the NAc mediate the active avoidance of aversive and punishing stimuli (Blomeley et al., 2018; Boschen et al., 2011; Danjo et al., 2014; Zalocusky et al., 2016), as well as clinical data that D2R availability is reduced in the ventral striatum of patients with SUDs (for review, see Volkow et al., 2009).

Greater clarity into the relationship between the NAc and risky reward seeking will likely be provided by more detailed analyses of the two anatomically and neurochemically dissociable NAc subregions, the NAcCore and NAcShell (Floresco, 2015; Zahm, 2000), although extant data to this end are conflicting. For example, pharmacological inactivations of the rat NAcCore produce a generalized reduction in the vigor of behaviors associated with reward (sucrose) seeking or punishment avoidance, without directly affecting risk seeking per se (Floresco et al., 2018; Piantadosi et al., 2018, 2017). Yet, in rodent models of quinine-resistant alcohol drinking, both N-methyl-D-aspartate (NMDA)-receptor modulation and chemogenetic inhibition of medium spiny neurons in the NAcCore can selectively reduce punished drinking, without generally affecting motivation (Seif et al., 2015, 2013; Sneddon et al., 2021). This pattern of results suggests that, depending on the nature of the reward (e.g., drug vs. natural reinforcer) or the type of functional manipulation, the NAcCore can be shown to promote or suppress reward-seeking behaviors.

Regarding the function of the NAcShell on negative punishment-based tasks, pharmacological inactivation has been shown to either decrease (Stopper and Floresco, 2011) or increase (Floresco et al., 2018) risky choices in rats. In contrast, inhibiting neural activity in the NAcShell generally increases risky behavior in mice and rats performing positive punishment-based assays, resulting in continued to reward seeking despite the prospect of footshock, for example (Halladay et al., 2020; Piantadosi et al., 2018, 2017). In rats, phasic DA release in the NAcShell, but not NAcCore, increases in response to punishing stimuli (Badrinarayan et al., 2012), and evoked DA release is positively correlated with preference for large, risky rewards (those associated with footshock punishment) (Freels et al., 2020).

In sum, the NAc contributes to multiple processes relevant to risky reward seeking. Still, much work remains to be done to dissect the precise functions of the NAc, with present data regarding the direct contributions of this region to reward seeking despite negative and positive punishment risk appearing largely equivocal. An important component of future investigations will be to examine the contributions afferent inputs to the NAc, such as those arising from the prefrontal cortex (PFC) and amygdala, to these behaviors.

Cortical contributions to risky decision-making

Neuropsychological studies of patients with focal brain lesions have established the orbitofrontal cortex (OFC) and medial PFC (mPFC) as important arbiters of decision-making in gambling tasks where risk is characterized by the potential for diminished or omitted reward (Bechara et al., 1999, 1994; Clark et al., 2008, 2003; Fellows and Farah, 2005). Patients with damage to either region perform more poorly on these tasks, not adapting their reward-seeking strategies despite mounting ‘losses’ and displaying minimal physiological responsivity in anticipation of reward (or punishment) related feedback (Bechara et al., 1999, 1994). These observations have led to hypotheses ascribing a role for these cortical regions in conveying emotional content to actions that have the potential to be punished (Bechara, 2000). Functional neuroimaging experiments in humans have provided insight into specific risk-related processes that alter metabolic activity in the OFC and ventromedial PFC, including uncertainty and arousal (Blankenstein et al., 2017; Critchley et al., 2001; Hsu et al., 2005; Huettel et al., 2006; Knutson et al., 2001; Tobler et al., 2007), punishment and devaluation (Gottfried et al., 2002; O’Doherty et al., 2001; Valentin et al., 2007), task-feedback after losses (Steward et al., 2019), and guesses (Bolla et al., 2004; Elliott et al., 1997). While there are differences in prefrontal anatomy between the human, non-human primate, and rodent brain (Heilbronner et al., 2016; Laubach et al., 2018; Wise, 2008), studies in laboratory animals have been valuable in helping define the roles of cortical structures to risky decision-making.

Orbitofrontal cortex: subjective outcome valuation and re-valuation

As in the case of midbrain DA neurons, it is likely that distinct subpopulations of OFC neurons encode specific aspects of task performance during risk seeking, such as reward or punishment probability, and the updating of these contingencies across trials (Hikosaka and Watanabe, 2000; Murray and Izquierdo, 2007; O’Neill and Schultz, 2018; Rolls et al., 1996; Scott et al., 2005; Simmons and Richmond, 2008; Tremblay and Schultz, 2000, 1999). For example, in non-human primates performing a delayed reaction time task, a majority of recorded OFC neurons encoded reward outcome expectation, with a subset of these neurons doing so in a reward-type specific manner (Hikosaka and Watanabe, 2000). Similarly, in a memory guided saccade task, a greater proportion of OFC neurons encoded expected reward value than probability of reward or risk (O’Neill and Schultz, 2018). Consistent with this idea, neuronal recordings, again in non-human primates, have revealed separate populations of OFC neurons that encode reward and risk predictions as a function of experience (O’Neill and Schultz, 2013; Raghuraman and Padoa-Schioppa, 2014).

Two main anatomical subdivisions of the rodent OFC have been implicated in processes relevant to risky decision-making: the lateral OFC (lOFC) and medial OFC (mOFC) (Fig. 2). To study OFC function in the context of loss-related punishment, work in rodents has often made use of behavioral assays that model human/non-human primate gambling tasks. For example, in vivo recordings performed in a rat version of the two-arm bandit task found that lOFC neurons encode choice outcome (i.e., reward value) in a manner that influences encoding and performance on subsequent trials (Sul et al., 2010). This finding suggests a role of the lOFC in updating the value of rewards following violations in outcome expectation; a notion consistent with lesion/inactivation studies showing the lOFC is critical for updating behavior following reward devaluation (Gallagher et al., 1999; Graybeal et al., 2011; Pickens et al., 2005; but see Ostlund and Balleine, 2007) and changes in stimulus-reward contingency (Dalley et al., 2004; Dalton et al., 2016; Groman et al., 2019; Izquierdo, 2017; Mobini et al., 2002; Pais-Vieira et al., 2007; Piantadosi et al., 2019; Ragozzino, 2007). Neuronal recordings in rats have revealed that lOFC activity tracks outcome value and uncertainty in alternative choice tasks that employ spatial (Feierstein et al., 2006; Roesch et al., 2006) or odor (Kepecs et al., 2008; Schoenbaum et al., 1998) cues, as well as violations of outcome expectation (in the form of errors) (Kepecs et al., 2008).

However, when explicitly examining negative punishment in the form of probabilistic discounting, decreasing neural activity within lOFC fails to alter behavior in well-trained rats (St. Onge and Floresco, 2010; Zeeb and Winstanley, 2011). This finding suggests that, although the lOFC may contribute to establishing or updating subjective biases (Miller et al., 2018; Mobini et al., 2002), it is not required for their maintenance once established. This seems to echo evidence that lOFC encoding of risk and reward outcome differs in individuals based upon subjective reward evaluation; risk-preferring rats exhibit increased lOFC neuronal activity on receipt of large rewards when obtained in risky, as compared to risk-free, trials (Roitman and Roitman, 2010). Interestingly, adolescent exposure to alcohol, which elevates risk-seeking behavior in adulthood, paradoxically diminishes the activity of a population of reward-encoding neurons in the lOFC of rats (McMurray et al., 2016). lOFC neurons have also been shown to strongly encode reward history on a probabilistic reward choice task, yet silencing this region optogenetically during the task epoch associated with reward-history related lOFC activity does not affect reward choice in rats (Constantinople et al., 2019a). Instead, optogenetic lOFC inhibition prior to choices decreased the pro-risk bias that ‘wins’ on preceding trials had on subsequent action selection (Constantinople et al., 2019a). Although these data suggest that lOFC neurons may track reward value in a manner dependent on risk preference, whether these neural correlates relate directly to choice bias remains unclear.

Surprisingly, there is a relative paucity of data on the role of the lOFC in reward seeking despite positive punishment. Increased neuronal excitability in the lOFC has been shown to correlate with resistance to footshock punishment in mice compulsively self-stimulating DA neurons (Pascoli et al., 2015). Tamping down lOFC excitation via lesions or chemogenetic has been found to decrease reward seeking in the face of footshock risk in both rats and mice (Orsini et al., 2015; Pascoli et al., 2015). Pharmacological inhibition of this region in rats has also been reported to reduce engagement in a cue-guided punishment task, which may reflect an enhanced sensitivity to positive punishment-induced avoidance (Verharen et al., 2019). These effects are suggestive of an impairment in outcome encoding following a loss of lOFC function, which would be broadly consistent with the findings from negative punishment-based tasks discussed above.

However, other studies are seemingly inconsistent with this view: in rats, lOFC inactivation can disinhibit footshock-punished reward seeking (Jean-Richard-dit-Bressel and McNally, 2016), while lesions of this region do not produce increases in cocaine seeking despite footshock punishment (Pelloux et al., 2013). Aside from technical differences stemming from the precise location targeted and degree of functional inactivation or compensation, these discrepancies argue against a simple unitary role for lOFC in encoding aversive outcomes to guide risky behavior. Instead, it is likely that heterogeneity within ensembles of lOFC neurons (based upon distinct anatomical location, inputs and outputs, and/or molecular characteristics) allows this region to provide complex task or context-specific control over reward-seeking behavior (for review, see Izquierdo, 2017). Providing support for the supposition that divergent lOFC outputs dissociably affect behavior, a recent study in rats found distinct functional roles for OFC neurons that project to the NAcCore or amygdala in utilizing positive or negative feedback, respectively, following reward expectation violations (Groman et al., 2019). Another study showed that depotentiation of dorsomedial striatum-projecting lOFC neurons in mice was sufficient to reverse resistance to footshock punishment in mice self-stimulating DA neurons (Pascoli et al., 2018). Developing a more granular picture of the functional circuity through which lOFC acts to affect punished decision-making should prove to be a fruitful avenue for work in the coming years.

Until recently, the contribution of mOFC to risky reward seeking has not been an area of intense study. Damage to the mOFC in humans and monkeys impairs value-guided decisions, resulting in reward-seeking actions that are relatively haphazard and loosely linked to their expected value (Noonan et al., 2017, 2010). In rodents, limited data suggest that the mOFC may oppose negative punishment-based risk seeking by lessening the impact of low probability ‘wins’ (i.e., unexpected reward delivery) on subsequent actions (Stopper et al., 2014a), though a null effect has been reported in a comprehensive study of mOFC function in economic choice (Gardner et al., 2018). In the latter study, optogenetic inhibition was ineffective at altering behavior when delivered at the onset of trials during which rats could choose between visual cues that predicted rewards of distinct magnitude, which in well-trained animals does not involve ‘risk’ per se (Gardner et al., 2018). Thus, the mOFC may contribute to risk processing operationalized as negative punishment. In support of this, recent data from a rat probabilistic discounting task showed that blocking D1-receptors in the mOFC reduced risk-seeking by enhancing the effect of losses on upcoming trials, while D2-receptor blockade tended to shift choice towards riskier options (Jenni et al., 2021).

In the context of positive punishment risk, data generally support a role for the mOFC in opposing risk-seeking that is comparable to negative punishment-based tasks. The ability to use a conditioned stimulus to inhibit actions that could incur footshock punishment is impaired following lesioning (Ma et al., 2020) or inactivation (Verharen et al., 2019) of the rat mOFC. Compulsive (footshock resistant) alcohol seeking in mice has also been associated with alterations in the excitability of mOFC neurons and the expression of genes for specific NMDA-receptor subunits (Radke et al., 2017). While limited in scope, these data together suggest that mOFC may generally function to oppose risk-seeking across positive and negative punishment-based designs, though the effect of local neuromodulation via DA may recruit subpopulations of neurons to produce distinct effects.

Medial prefrontal cortex: top-down inhibition and updating biases

In humans, the anterior cingulate cortex (ACC) and dorsolateral PFC (dlPFC) are suggested to support decision-making via their contribution to enacting executive functions such as inhibitory control (Bembich et al., 2014; Manes et al., 2002). This view aligns well with the findings of human neuroimaging studies demonstrating activation of the dlPFC and ACC during probabilistic decision-making as assessed by the Iowa Gambling Task (IGT), which requires participants to select from decks of cards associated with hidden probabilities of reward and punishment, using feedback from previous selections to avoid losses and maximize gains (Bolla et al., 2005; Ernst et al., 2003, 2002; Li et al., 2010; Mohr et al., 2010). Neuropsychological studies support a causal role for dlPFC activity on the IGT and similar laboratory assays of negative punishment, with patients harboring dlPFC lesions making riskier choices (i.e., opting for low probability, high value options versus high probability, low value options) than non-lesioned participants (Clark et al., 2003; Fellows and Farah, 2005; Manes et al., 2002).

Bearing in mind the significant differences in cortical anatomy across species, many studies have demonstrated that the rodent mPFC mediates adaptive behavior following the introduction of probabilistically-delivered positive punishment (Halladay et al., 2020; Jacobs and Moghaddam, 2020; Orsini et al., 2018) or reward omission (Passecker et al., 2019; Rivalan et al., 2011; St. Onge and Floresco, 2010) on reward-seeking tasks (Fig. 2). In the context of loss-related negative punishment, disrupting neural transmission in the mPFC renders rats insensitive to reward contingency shifts (Balleine and Dickinson, 1998) and can impair the normal evaluation of risky actions (Jentsch et al., 2010; Paine et al., 2015, 2013; Piantadosi et al., 2016; St. Onge et al., 2012b; St. Onge and Floresco, 2010; van Holstein and Floresco, 2020; Zeeb et al., 2015). Inactivation of the mPFC prevents rats from updating their choice bias adaptively during risky negative punishment tasks (Jentsch et al., 2010; St. Onge and Floresco, 2010).

Performance on positive punishment-based tasks in rodents also appears to require mPFC activity. For instance, in vivo recordings of calcium activity indicate that mPFC neuron activity adapts as rats learn about the risk of a footshock punishment, such that the phasic inhibitory responses that occur during the execution of risky actions early in learning are diminished late in learning (Jacobs and Moghaddam, 2020). These correlative responses likely reflect a causal contribution of the mPFC, given that inhibiting neural activity in this region can result in rat reward-seeking behavior that is indifferent to the risk of positive punishment (Friedman et al., 2015; Orsini et al., 2018; Resstel et al., 2008; but see Jean-Richard-dit-Bressel and McNally, 2016).

This literature is generally in keeping with the commonly held view that mPFC is recruited to inhibit risky reward seeking, and that this function may be diminished in addiction (Hu et al., 2019; Lüscher et al., 2020; Smith and Laiks, 2018). For example, silencing rat mPFC neurons promotes, while activating this region attenuates, cocaine-seeking behavior in the face of potential footshock (Chen et al., 2013). It should be noted that the role of mPFC in regulating punished reward seeking is likely pathway dependent. In the context of alcohol seeking, silencing mPFC neurons projecting to the NAcShell increases alcohol seeking in mice that would otherwise be inhibited by a prior experience with response-contingent footshock punishment (Halladay et al., 2020). Similar optogenetic inhibition of a mPFC input to the midbrain periaqueductal gray (PAG) increases risky alcohol seeking in mice, possibly by preventing the aversive properties of the punishing stimulus from being encoded (whether footshock or the bitter taste of quinine) (Siciliano et al., 2019). In contrast, optogenetic inhibition of mPFC (or insular cortex) projections to the NAcCore has been shown to decrease risky alcohol seeking in rats (Seif et al., 2013).

Another factor that could contribute to heterogeneity in risk related mPFC function is the existence of prefrontal subregions, including the prelimbic (PL; functionally analogous to area 32 of the human ACC) and infralimbic (IL; akin to human ACC area 25) cortices, that are at least partially anatomically and functionally distinct (Passecker et al., 2019; van Holstein and Floresco, 2020; but see Zeeb et al., 2015; Moorman et al., 2015). In this context, while neurons in both the IL and PL are activated by a positive punishment-based risky decision-making task in mice, only IL activity correlated with lower levels of risk-taking (Glover et al., 2020). Likewise, subregion-specific recordings have found a population of neurons in the IL, but not PL, that encode alcohol-seeking behaviors in mice prior to footshock punishment, but this response disappears in favor of signaling behaviors associated with the punishment-induced inhibition of alcohol seeking (Halladay et al., 2020).

Alterations in synaptic plasticity and the excitability of PL pyramidal neurons have also been correlated with resistance to punishment in rats self-administering cocaine (Chen et al., 2013; Kasanetz et al., 2013). Within this same region, neurons encode outcomes associated with distinct probabilities of reinforcement, and guide upcoming actions based on prior feedback (Passecker et al., 2019). Optogenetic silencing of rat PL neurons (Passecker et al., 2019) or mouse NAcShell-projecting IL neurons (Halladay et al., 2020), has been found to increase risky actions, consistent with work demonstrating that pharmacological inactivation of either PL or IL produces insensitivity to cued punishment, perhaps by impairing the ability to use the learned cue-punish association to inhibit behavior (Verharen et al., 2019). These data argue for some degree of functional overlap between rodent mPFC subregions, which could be accomplished by distinct neuronal populations that project to separate downstream regions, but exist across the traditional mPFC subregion boundaries (Moorman et al., 2015).

Given the importance of DA to risky decision-making, it is noteworthy that the mPFC is reciprocally connected to the midbrain, and punishment or aversive events can induce changes in the dynamic communication between VTA cells and a dorsal mPFC region encompassing the PL in rats (Park and Moghaddam, 2017; Vander Weele et al., 2018). Furthermore, alcohol exposure causes dendritic abnormalities in the mouse mPFC (Jury et al., 2017) and alterations in markers for DA synthesis in the rat mPFC that are associated with increased risk preference (Boutros et al., 2015). Microinfusion of a D1R antagonist (SCH23390) or D2R agonist (quinpirole) into the IL of rats opposes footshock-punished reward seeking and promotes goal-directed behavior in the absence of punishment, suggesting prefrontal DA modulates adaptive behavior following changes in action-outcome contingency (Barker et al., 2013). This effect is similar to the blockade of mPFC D1Rs on a probabilistic discounting task in rats, which results in less risk seeking as a result of enhancing the impact of losses on future behavior (St. Onge et al., 2011).

DA can also affect risk seeking in distinct ways depending on the specific mPFC output pathway affected. For example, disrupting D2R-based modulation of mPFC neurons prevented rats from updating behavior following losses via output to the amygdala, while disrupting D1R-based mPFC modulation increased risky actions via projections to the NAc (Jenni et al., 2017). This dissociation ties in with growing evidence that the mPFC regulates footshock-punished reward seeking via discrete output pathways: mouse mPFC neurons projecting to the NAc, but not VTA, have been implicated in inhibition of punished reward seeking (Halladay et al., 2020; Kim et al., 2017). This raises a pertinent corollary question regarding what role, if any, mPFC inputs to the VTA play in regulating punished behaviors (Jhou and Vento, 2019; Verharen et al., 2020a). Overall, these data establish mPFC as a region that can regulate risky reward seeking largely via the top-down regulation of inhibitory control and behavioral flexibility. Identifying how specific subregions and their outputs contribute to distinct domains of risk seeking remains an area of active study.

Amygdala: encoding punished outcomes to guide future actions

The amygdala shares reciprocal projections with the OFC and mPFC, receives input from midbrain DA neurons, and innervates the NAc, factors which make this region particularly well positioned to regulate risky decision-making (Fig. 2). Indeed, amygdala interactions with cortical areas are known to mediate processes that are likely key to regulating behavior in the face of potential punishment across species, including the encoding of reward values (Malvaez et al., 2019; Rudebeck et al., 2013) and the updating of information about stimulus/response-outcome relationships following contingency shifts (Baxter et al., 2000; Burgos-Robles et al., 2017; Fiuzat et al., 2017). Studies in rodents have shown that the utilization of outcome-related information to guide action selection in risky contexts is critically dependent on the amygdala (Bercovici et al., 2018; Ghods-Sharifi et al., 2009; Jean-Richard-Dit-Bressel and McNally, 2015; Malvaez et al., 2019; Orsini et al., 2015; Piantadosi et al., 2017; Zeeb and Winstanley, 2011). Loss of these functions could explain the observation that humans with amygdala lesions exhibit suboptimal patterns of choice on gambling tasks and blunted autonomic responses to wins and losses indicative of impaired affective engagement (Bechara et al., 1999; Gupta et al., 2011).

Among the amygdala subnuclei pertinent to risky reward seeking, studies in rodents have consistently implicated the basolateral nucleus of the amygdala (BLA). Plasticity within this nucleus is critical to various forms of aversive learning, including Pavlovian fear and instrumental avoidance, both of which are impaired by manipulations that decrease BLA activity (Bravo-Rivera et al., 2014; Choi et al., 2010; Fanselow and LeDoux, 1999; Lázaro-Muñoz et al., 2010; Ramirez et al., 2015; Sengupta and Holmes, 2019). Given that positive punishment necessitates the integration of a primary aversive stimulus (e.g., footshock) with reward-seeking motivation, it is unsurprising that BLA activity is necessary for risk to inhibit reward seeking or induce choice flexibility (Ishikawa et al., 2020; Jean-Richard-Dit-Bressel and McNally, 2015; Orsini et al., 2015; Pelloux et al., 2013; Piantadosi et al., 2017). For example, pharmacological inactivation of BLA causes rats to continue to engage in reward-seeking behavior despite footshock punishment (Jean-Richard-Dit-Bressel and McNally, 2015; Piantadosi et al., 2017), and lesions of this nucleus prevent rats from shifting to reward options that are not associated with footshock risk (Orsini et al., 2015). This function extends to drug-seeking contexts, as BLA lesions have been shown to produce compulsive, footshock resistant cocaine-seeking in rats with extensive self-administration experience (Pelloux et al., 2013). Based on these results, BLA appears to be recruited to inhibit reward-seeking in the context of positive punishment.

In contrast to the risk-avoiding role of the BLA indicated by these studies, evidence suggests that the BLA typically promotes reward seeking in the face of negative punishment (Ghods-Sharifi et al., 2009; Larkin et al., 2016; Tremblay et al., 2014; but see Zeeb and Winstanley, 2011). Pharmacological inactivation of the rat BLA was shown to enhance loss-sensitivity, decreasing the selection of a preferred, but risky, action in favor of a less-preferred safe action (Ghods-Sharifi et al., 2009). Similarly, BLA lesions decreased the willingness of rats to ‘chase losses’, which refers to the preference to continue gambling to avoid a small loss at the risk of larger future losses (Tremblay et al., 2014). Blockade of D1Rs, receptors which natively function to increase the excitability of BLA neurons and promote reward seeking (Di Ciano and Everitt, 2004; Kröner et al., 2005), also decrease risky decisions in rats by diminishing the impact of wins (i.e., reward delivery despite low probability) on subsequent choices (Larkin et al., 2016). The BLA likely interacts with a distributed cortico-limbic-striatal network to modify behavior in the face of negative punishment risk. Pharmacological disconnections in rats performing a probabilistic discounting negative punishment task suggested that the BLA may interact with the NAc to promote risky actions despite loss, while top-down inhibition of this nucleus by the mPFC may be necessary to temper the drive towards preferred rewards mediated by the BLA (St. Onge et al., 2012b).

One apparent conclusion from these primarily pharmacological experiments is that the BLA fulfils opposing roles in modifying reward seeking depending on whether punishment is positive or negative: biasing mice away or towards risky options, respectively. Providing nuance to this generalization, recent optogenetic experiments have begun to reveal distinct contributions of BLA to particular phases of the risky decision-making process (Orsini et al., 2019). One such study optogenetically inhibited the BLA of rats during either the pre-choice deliberation phase or the outcome encoding phase of a footshock-punished decision-making task and found that post-choice silencing increased, while pre-choice silencing decreased risk-taking (Orsini et al., 2017). The post-choice silencing effect fits well with the view that the BLA encodes aversive outcomes to guide future actions; hence, the loss of BLA function during this epoch would make rats effectively ‘blind’ to the introduction of punishment and maintain reward seeking despite the potential harm (Jean-Richard-Dit-Bressel and McNally, 2015; Piantadosi et al., 2017). However, the finding that BLA inhibition timed to the pre-choice epoch decreased footshock-punished risk seeking is harder to reconcile with prior pharmacological experiments. One parsimonious explanation for this apparent disconnect is that BLA activity prior to choices promotes the selection of preferred rewards across domains of risky decision-making. Partial support for this hypothesis has been provided by a study that used pre-choice optogenetic silencing of rat BLA projections in the NAc to reduce responding for the preferred large (but uncertain) reward in a negative punishment task, although this bias was only present when the probability of reward was high (when the chance of reward was low, silencing instead increased risk-taking) (Bercovici et al., 2018).

Although the BLA is of clear importance to utilizing the experience of punished outcomes to adjust behavior, exactly how this region contributes to deliberations about risky actions remains a key question that is yet be fully parsed. Part of the answer likely lies in task-phase specific functional distinctions between BLA outputs to subregions of the NAc and other projection targets, including the central amygdala and ventral hippocampus, which likely represent valence in distinct ways (Beyeler et al., 2018, 2016). Dissecting these pathways in the context of risky decision-making should provide a rich seam for future studies to mine. Another key question relates to how BLA produces opposite patterns of behavior following reward omission or positive punishment. A recent investigation in rats provided compelling evidence that footshock induced suppression of cocaine seeking is disinhibited by BLA inactivation, while reward omission (extinction) induced suppression of cocaine seeking is inhibited by BLA inactivation (Pelloux et al., 2018). These data, in combination with the studies highlighted above, suggest a fundamental distinction in how BLA encodes positive versus negative punishment to affect reward seeking. Future experiments using projection-specific (e.g., calcium-imaging based) measurement of BLA neuron activity during risky reward-seeking may help disentangle the specific neural ensembles recruited by particular types of aversive outcomes.

Conclusions and key questions for the future

Here we have reviewed a host of data which establishes meso-cortico-limbic-striatal brain regions as integral nodes in various domains of risky reward-seeking behavior. While our functional understanding of these regions remains incomplete, a broad picture of their contributions has begun to emerge. The activity of DA neurons of the VTA are tightly regulated by upstream regions, enabling these neurons to signal aspects of incentive motivation and prediction error via output to targets including the NAc. Cortical regions also receive DA input, which likely influences the ability of subregions like the OFC to generate predictions about outcomes and the mPFC to maintain choice flexibility. Finally, the amygdala is necessary to update actions based on the risk inferred from the occurrence of salient outcomes, including wins and punishments.

Although this picture of circuit function is compelling, there are critical gaps in our understanding that have only begun to be addressed. One such question relates to the relevance of the recently appreciated diversity of VTA neuron composition and function (Morales and Margolis, 2017; Verharen et al., 2020b). For example, subtypes of VTA neurons can co-release DA and amino acid transmitters like glutamate within the NAc to mediate aversion (Qi et al., 2016; Yamaguchi et al., 2015; Zhang et al., 2015). Other non-canonical VTA neurons appear to release GABA and/or glutamate in structures known to regulate risk seeking such as the LHb and mPFC (Kabanova et al., 2015; Root et al., 2014; Stamatakis et al., 2013).

Even within ‘classical’ VTA DA neurons, DA release in terminal regions such as the mPFC and NAcCore and NAcShell is evoked by stimuli of distinct valence (de Jong et al., 2019; Kim et al., 2017; Lammel et al., 2011). Phasic DA release in the medial NAcShell is increased by aversive stimuli and the cues that predict them, whereas DA release in other regions of the NAc is generally decreased in response to the same stimuli (Badrinarayan et al., 2012; de Jong et al., 2019). Importantly, such heterogeneity may be the rule, rather than the exception, across the neural landscape. For example, molecularly distinct projection populations that contribute differently to positive and negative valence behaviors exist in the BLA (Beyeler et al., 2018, 2016; Kim et al., 2017; Shen et al., 2019) and other amygdala subregions (Hardaway et al., 2019; Janak and Tye, 2015; Miller et al., 2019). These data speak to functional diversity at the level of a given brain region, characteristics which have yet to be thoroughly investigated in the realm of risky reward seeking.

Related to such functional diversity, developing a map of the neural mechanisms contributing to risky actions necessitates an appreciation for the multiple convergent sub-processes that contribute to action selection (Orsini et al., 2019). Risky decisions can be conceptualized as being comprised of at least two distinct phases: the pre-choice deliberation phase and the outcome/feedback phase. Optogenetic techniques provide the ability to delineate the function of these epochs by enabling temporally precise control of neuronal excitability. Indeed, recent data have demonstrated that contributions to these distinct task epochs may be dissociable, even within a given brain region (Bercovici et al., 2018; Constantinople et al., 2019a; Friedman et al., 2015; Hernandez et al., 2019; Li et al., 2019; Orsini et al., 2017; Stopper et al., 2014b). This complex regulation may have important implications for the implementation of therapeutics designed to improve maladaptive decision-making. Treatments that monotonically affect regions or circuits may not be sufficient to reverse decision-making deficits unless they are delivered during particular phases of the decision-making process (Adams et al., 2017; Orsini et al., 2019). More detailed analyses in model systems using temporally precise manipulations to dissect component processes of decision-making will be useful in this regard.

In conjunction with epoch-specific analysis, there is a need to continue to move towards understanding the pathways and circuits that contribute to risky actions, rather than single brain regions. Many of the regions highlighted here are at most one or two synapses upstream or downstream from each other, leaving little doubt that functional connectivity between these nodes is critical. Techniques to identify inputs and outputs of discrete genetically-defined populations (e.g., Schwarz et al., 2015) and to trace and transsynaptically manipulate specific pathways (e.g., Zingg et al., 2020, 2017) provide avenues to investigate the connectomics of risky reward seeking. For example, in rats, the BLA contributes to aspects of risky decision-making via input to the NAc (Bercovici et al., 2018), yet it is unclear what influence this excitatory afferent has on NAc activity during discrete decision-making epochs to affect behavior. Additionally, BLA (like many cortico-limbic regions) sends projections to both the NAcShell and NAcCore, with these anatomically segregated pathways potentially contributing differentially to the inhibition and invigoration of behavior, respectively (Fig. 2; Cisneros-Franco and de Villers-Sidani, 2018; Folkes et al., 2020; Millan et al., 2017; Piantadosi et al., 2017; Stuber et al., 2012). Evaluating the function of these and other subregion-specific pathways may provide clarity to the sometimes-conflicting findings observed across positive and negative punishment tasks.

Finally, while compulsive behaviors in AUDs and SUDs incur consequences that can be considered punishers, such as job loss or health complications, these events are rarely experienced immediately on drug seeking or taking, rather, they are often delayed or accumulate on long timescales. This contrasts with the situation in a typical rodent behavioral assay, where the punishment is temporally proximal to the reward-seeking action(s) (Fig. 1C) and has led to reasonable criticisms of the translational significance of the pre-clinical paradigms (for expanded discussions, see Epstein and Kowalczyk, 2018; Vanderschuren et al., 2017). Perhaps many of the paradigms described in this review may be most relevant to individuals considering or seeking treatment for their addiction, given that in these cases there is a clear recognition of the threat of punishment, even if remote. Nonetheless, in recognition of the potential distinction between delayed and immediate punishment, assays of delayed punishment for use in animals have been developed. Delayed punishment has been shown to reduce cocaine self-administration in monkeys, albeit less effectively than immediate punishment (Woolverton et al., 2012). Similarly, recent work has shown that delayed punishment is less effective than immediate punishment at affecting subsequent risky choices, with this effect being more prominent in male rats than female rats (Liley et al., 2019; Rodríguez et al., 2018). This preliminary work sets the stage for future studies to further develop and characterize delayed punishment paradigms and assess the extent to which they may recruit neural mechanisms distinct from those involved in the effects of immediate punishment.

Ultimately, the data reviewed here provide evidence that a distributed set of interconnected brain regions contribute to risky reward seeking. The literature paints a picture of both shared and divergent neural regulation of risky reward-seeking behaviors, depending on the way in which risk is operationalized. Critically, dysfunction within many of these neural structures has been linked to disorders characterized by inappropriate behavior in the face of negative consequences, including AUDs, SUDs, and behavioral addictions. The use of cross-species assessments employing cutting-edge optical and viral techniques will continue to refine our understanding of the mechanisms contributing to reward seeking despite negative and positive punishment. These pre-clinical investigations have the potential to improve the lives of individuals suffering from these disorders by identifying mechanisms that may be exploited in the future to develop novel therapeutics.

Abbreviations

ACC

anterior cingulate cortex

AUD

alcohol use disorder

BLA

basolateral amygdala

D1R

dopamine D1-receptor

D2R

dopamine D2-receptor

DA

dopamine

dlPFC

dorsolateral PFC

GABA

gamma-aminobutyric acid

IGT

Iowa Gambling Task

IL

infralimbic cortex

LHb

lateral habenula

lOFC

lateral orbitofrontal cortex

mOFC

medial orbitofrontal cortex

mPFC

medial prefrontal cortex

NAc

nucleus accumbens

NAcCore

nucleus accumbens core

NAcShell

nucleus accumbens shell

NMDA

N-methyl-D-aspartate

OFC

orbitofrontal cortex

PAG

periaqueductal gray

PFC

prefrontal cortex

PL

prelimbic cortex

RMTg

rostromedial tegmental nucleus

RPE

reward prediction error

SUD

substance use disorder

VTA

ventral tegmental area