Neurobiology of Reward-Related Learning

Abstract

A major goal in psychology is to understand how environmental stimuli associated with primary rewards come to function as conditioned stimuli, acquiring the capacity to elicit similar responses to those elicited by primary rewards. Our neurobiological model is predicated on the Hebbian idea that concurrent synaptic activity on the primary reward neural substrate—proposed to be ventral tegmental area (VTA) dopamine (DA) neurons—strengthens the synapses involved. We propose that VTA DA neurons receive both a strong unconditioned stimulus signal (acetylcholine stimulation of DA cells) from the primary reward capable of unconditionally activating DA cells and a weak stimulus signal (glutamate stimulation of DA cells) from the neutral stimulus. Through joint stimulation the weak signal is potentiated and capable of activating the VTA DA cells, eliciting a conditioned response. The learning occurs when this joint stimulation initiates intracellular second-messenger cascades resulting in enhanced glutamate-DA synapses. In this review we present evidence that led us to propose this model and the most recent evidence supporting it.

Introduction

In psychology the concept of associations is central to understanding learning and the organization of behavior. The notion that psychological elements come to be associated by mental processes can be traced to Aristotle who, himself, proposed the mechanisms of frequency and similarity in the formation of associations (Barnes, 2001; Warren, 1921). So important was associationism to philosophers that Hobbes and other empiricists considered it to be one of only several, if not the only, mental process of the mind (Hergenhahn, 2005). The concept of association refers to the observation that sensations, thoughts, emotions and actions appear to be connected to each other. Pavlov believed his demonstrations that neutral stimuli can acquire the capacity to elicit physiological responses like those elicited by the naturally occurring stimuli were the long-awaited empirical demonstrations of what philosophers had long referred to as mental associations (Pavlov, 1960, 1928). Thorndike, Watson and Skinner focused much of their work on the learning of associations as they set out to study and explain the organization of behavior. The idea that psychological elements form associations through materialistic connections can be traced back to at least Alexander Bain, who proposed that psychological elements become associated when they are laid down in the brain together as a package (Bain, 1977, 1875). Thorndike proposed that the underlying mechanism of the Law of Effect is the formation of a neural bond between the elements (Thorndike, 1970, 1911). Although Thorndike had set the goal of studying this neural bond as part of his research aims he did not achieve this goal. The discovery of the “neural bond” would have to await the latter half of the 20^th century with the theoretical propositions of Donald O. Hebb and the fruits of subsequent research into his propositions.

Hebb’s theoretical contribution to our understanding of the neural mechanisms underlying learning arises at least partly out of the frustrations of his mentor, Karl Lashley. Lashley proposed that memories were situated in the brain as engrams, each of which represented a memory and resided in a specific part of the brain (Lashley, 1950). Lashley’s many years of research produced several groundbreaking findings, such as mass action and equipotentiality (Lashley, 1929; Nadel and Maurer, 2018), but his research did not find evidence to support the idea of the region-specific engram. Equipped with this knowledge and the observations he made with Wilder Penfield at the Montreal Neurological Institute while evaluating recovering neurosurgery patients (Hebb and Penfield, 1940), Hebb proposed a theory of how the brain organizes information. His theory revolutionized physiological psychology and set into motion decades of ongoing neuroscience research uncovering the mechanisms underlying learning. At the heart of Hebb’s theory is that synaptic activity occurring while the postsynaptic neuron is active (e.g., firing action potentials) results in a change in the strength of the synapse (Cooper, 2005; Hebb, 1949). Thus, Hebb proposed a model whose components were the neural correlates of the psychological elements about to form an association; element 1: the synaptic activity, element 2: the active postsynaptic neuron, the association: the changed synaptic strength (Cooper, 2005; Hebb, 1949). Could this strengthened synapse represent the association between psychological elements? Could it be the “neural bond” that Thorndike considered? Modern neuroscience research on learning is a continuation of the research sparked by Hebb’s model.

In the 1960’s and early 1970s Lømo and Bliss provided evidence supporting Hebb’s view of synaptic change with their discovery of long-term potentiation, known as LTP. While conducting electrophysiological experiments to investigate the role of the hippocampus in short-term memory, Lømo unexpectedly observed that electrical stimulation to the presynaptic fibers at high frequencies elicits stronger and prolonged excitatory postsynaptic potentials (EPSP) in the postsynaptic neurons, when subsequently subjected to single-pulse stimuli (Bliss and Collingridge, 1993; Bliss et al., 2003; Bliss and Lømo, 1973; Lømo, 1966). Later, as the properties of LTP were studied, it was also observed that potentiation of a weak synapse is obtained when that weak synapse is active while an accompanying strong synapse is activating the post-synaptic neuron (Barrionuevo and Brown, 1983; Bliss and Collingridge, 1993; Humeau et al., 2003; Kandel et al., 2013; Kelso and Brown, 1986; Levy and Steward, 1983, 1979; Martinez et al., 2002) [(for review see (Hao et al., 2018; Nicoll, 2017; Nicoll et al., 1988)]. This latter form is referred to as associative LTP and thought to be an underlying physiological mechanism of associative learning.

This review will focus on a specific type of associative learning – reward-related learning. We propose a Hebbian type neurobiological model of reward-related learning that attempts to explain learned responses to stimuli that have been associated with primary reward. Understanding this type of reward-related associative learning is critical for understanding reward-driven behavior, referred to as incentive motivation, as well as pathologies such as drug addiction and alcoholism, which have at their core, maladaptive reward-related learning. Note, the model we propose is not intended as a model of associative learning in general. Rather, this model is intended to explain how reward-associated stimuli come to acquire rewarding properties of their own – in the way that a heroin cue can act as a reward itself – which we believe involves associative learning mechanisms. Further, we understand that the Hebbian model may not account for all types of associative learning and we would not like to make the case that it does. The Hebbian model – neurons that fire together, wire together – however, does appear to account for the types of reward-related learning that we review here.

The function of conditioned stimuli associated with reward

Environmental stimuli that are paired with rewards can influence behavior and serve different behavioral functions. These stimuli can provide information about where, when and how to obtain rewards; therefore they can acquire informative/predictive functions that enable survival and or adaptation to new environments. Stimuli paired with reward can come to function as conditioned stimuli (CS) that elicit responses similar to those elicited by primary rewards (US); in this case the responses are referred to as conditioned responses (Pavlov, 1928). The stimuli perceived prior to consumption of drugs of abuse (e.g., drug paraphernalia such as syringes, powders) can elicit drug craving and seeking in addicted individuals. CSs that signal the availability of rewards can increase motivation to work for them or engage in similar behaviors that are reinforced by these CSs, which in this case are also functioning as conditioned reinforcers (CRs). For example, rats will readily press a lever for a stimulus (CS) that was previously paired with brain stimulation reward (Stein, 1958) or food (Beninger and Ranaldi, 1992; Ranaldi and Beninger, 1993) and some addicted individuals will engage in needle fixation behavior when lacking heroin (Treffurth and Pal, 2010). Given that a CS has multifaceted adaptive functions and plays a critical role in the maintenance of maladaptive behaviors, it is important to understand how environmental stimuli come to function as CSs that control reward-driven behaviors (i.e., produce conditioned responses). We propose that, because cognitively and behaviorally, the CS becomes associated with the primary reward and with the unconditioned response normally produced by the primary reward, on a neural level and based on Hebb’s model and what we know about associative LTP, a similar thing happens at the neural level; the neural representation of the CS becomes synaptically connected (i.e., associated) with at least part of the same circuit that mediates the unconditioned effects of the primary reward with which it is contiguously paired.

Note that we are not suggesting that CSs substitute for USs, that they come to take on all the neural and psychological properties of USs and elicit an identical response to the unconditioned response. CSs can also acquire anticipatory and or predictive properties (Holland, 1977; Timberlake and Grant, 1975). For instance, in our conditioned approach paradigm, different responses/behaviors such as orienting toward the light, approaching the food trough, consuming the food pellet, and others, that occur during and just after the presentation of the light or tone CSs might be adventitiously reinforced and emerge as a consequence of the past reinforced experiences. In the same paradigm, the light and tone CSs, because of contiguity with the food pellets (US), can also acquire the capacity to produce effects similar to those produced by ingestion of the food pellets which will include gustatory sensations (i.e., tastes), pleasure or “liking”, priming or incentive motivation, anticipation and other cognitive, motivational and physiological effects. We suggest that CSs, by association with the US and the unconditioned responses elicited by the US, acquire the capacity to elicit neural and behavioral responses related to incentive motivation for the primary reward as well as a subset of the neural and behavioral responses in the repertoire of unconditioned responses (e.g., reinforcement) naturally elicited by the primary reward.

1. Our neurobiological model of reward-related learning

In the 2000’s we proposed that stimuli associated with primary reward come to function as CSs because they acquire the capacity to activate parts of the neural system that are naturally activated by the primary rewards themselves with which they are paired (Ranaldi, 2014; Sharf et al., 2006; Sharf and Ranaldi, 2006; Zellner and Ranaldi, 2010). This is an idea that had been proposed previously (Bindra, 1974; Schultz and Dickinson, 2000). In so doing, CSs therefore elicit behaviors that are similar to those elicited by the primary reward because such stimuli activate parts of the same neural circuit, producing the behaviors, as the primary reward. In this review we will lay out this model, lay out the neural components of this model, propose the types of neural events that would be necessary for associative reward-related learning to occur under this model, expand the model to incorporate new findings and provide new evidence in support of this model.

1.1. Unconditioned primary reward and reward-associated conditioned stimuli activate DA systems

It has long been established that dopamine (DA) neurons in the ventral tegmental area (VTA) play a strong role in mediating the rewarding effects of natural and artificial rewards (Berridge and Robinson, 1998; Wise, 2004; Wise et al., 1978; Wise and Rompre, 1989). Food, water, sex, social interaction and drugs of abuse are known for their ability to activate VTA DA neurons (Hansen and Whishaw, 1973; Hernandez and Hoebel, 1988; Wise et al., 1978) and lead to phasic DA release in the nucleus accumbens (NAc) (Cheer et al., 2007, 2004; Di Chiara and Imperato, 1988, 1986; Rada et al., 2000). Animals with severe depletion of DA will reject food or water and will die of starvation, if not fed artificially (Anand and Brobeck, 1951; Marshall and Teitelbaum, 1973; Stricker et al., 1978; Ungerstedt, 1971; Zigmond and Stricker, 1972). Similarly, mice with genetic deletion of DA neurons will survive when artificially fed with Ensure Plus or given L-DOPA (a DA synthesis precursor) treatment (Szczypka et al., 1999), suggesting that the DA system is critically involved in mediating the effects of primary rewards. The same DA system has been implicated in drug rewards. Opioids are thought to excite VTA DA neurons by inhibiting local GABA or GABA afferents, resulting in DA firing and release (Johnson and North, 1992; Jhou et al., 2009; Matsui and Williams 2011; Matsui et al., 2014). Morphine is self-administered directly into the VTA (Bozarth and Wise, 1981; David and Cazala, 1994; Devine and Wise, 1994; Welzl et al., 1989) and intra-VTA infusions of opioid agonists can lead to the acquisition of conditioned place preference (CPP) (Bari and Pierce, 2005; Caine et al., 1995; Phillips et al., 1994). In addition, systemic administration of DA receptor antagonists attenuates cocaine, amphetamine, heroin or oxycodone reward, as assessed in various drug-self administration paradigms (de Wit and Wise, 1977; Galaj et al., 2016, 2014; Song et al., 2011; You et al., 2018). Similar effects have been observed after intra-NAc injections of DA antagonists (Bari and Pierce, 2005; Caine et al., 1995; Phillips et al., 1994) or optogenetic inhibition of VTA DA neurons (Corre et al., 2018; Galaj et al., 2020), suggesting that the mesolimbic DA system is critically involved in drug reward.

It is also well-established that stimuli that are associated with primary rewards (e.g., drug paraphernalia) come to possess the ability to activate the mesocorticolimbic DA system to some degree. The presentation of CSs is associated with increased calcium transients (an indicator of neural activation) of VTA DA cells (Saunders et al., 2018), increased VTA DA cell firing (Kiyatkin and Rebec, 2001; Kosobud et al., 1994; Miller et al., 1981; Pan et al., 2005; Pan and Hyland, 2005; Schultz, 1997), and the release of DA in the NAc (Gratton and Wise, 1994; Ito et al., 2002; Kiyatkin and Gratton, 1994; Phillips et al., 1983; Stuber et al., 2008). Importantly, this conditioned VTA DA neuronal activity subserves the expected corresponding behavioral functions that would be associated with CSs. Excitotoxic lesions of NAc impair conditioned approach responding elicited by a food-associated CS (Parkinson et al., 1999) or self-administration maintained by presentation of drug-associated cues (Ito et al., 2004). Intra-NAc infusions of DA D1 or D2 receptor antagonists reduce responding maintained by food or drug-related CSs (Di Ciano et al., 2001; Nicola et al., 2005; Wakabayashi et al., 2004; Yun et al., 2004). Thus, CSs can cause the conditioned activation of VTA DA neurons and this neural activity appears necessary for CSs to function as such.

Our group has shown that when a light stimulus is explicitly paired with reward, the light stimulus acquires the ability to elicit conditioned activity in DA cells of the VTA (measured as cfos in tyrosine hydroxylase labeled cells) (Galaj and Ranaldi, 2018; Kest et al., 2012). This finding concurs with findings described above showing that CSs increase VTA DA cell activity. Importantly, the same light stimulus also functions as a CS; it elicits conditioned approach. Therefore, similar to how a reward-associated CS acquires the capacity to elicit a reward-related response that is similar to the unconditioned response, the reward-associated CS also acquires the capacity to elicit at least part of the neural response—activation of VTA DA neurons—that underlies the primary reward response. In this case, a neural association between the reward-associated stimulus and the activation of VTA DA neurons (i.e., reward-related response) is formed. Our model is an attempt to explain how this neural association is formed. That is, our model is an attempt to describe how a previously neutral stimulus acquires the capacity to activate VTA DA neurons and consequently function as a CS (and conditioned reinforcer).

As we mentioned above CSs acquire the capacity to activate a subset of the neural and behavioral responses in the repertoire of unconditioned responses possessed by primary rewards including the capacity to produce reinforcement, which in this case is referred to as conditioned reinforcement/reward. Also as we describe above, primary rewards derive their rewarding/reinforcing properties at least partly from their ability to cause or facilitate DA neurotransmission in the terminal regions of the VTA DA cells, and specifically in the NAc. We propose that CSs derive their capacities to produce conditioned reward or reinforcement through their acquired ability to activate VTA DA neurons resulting in similar enhanced DA neurotransmission in terminal regions.

1.2. The components of a basic Hebbian type model

We propose that a reward-associated CS comes to function as such because it acquires the capacity to activate parts of the neural system typically activated by the natural reward (see Figure 1). Thus, a Hebbian type model would stipulate that the primary reward neural system (re: VTA DA neurons) receive both a strong US signal from the primary reward capable of unconditionally activating it and a weak stimulus signal from the originally neutral reward-associated stimulus that is originally too weak to activate the reward system. With joint stimulation the weak signal becomes potentiated and can activate the reward neural system (VTA DA cells) (although to a lesser degree than the natural rewards), enabling the stimulus to now function as a CS and conditioned reward. In a series of papers (Galaj et al., 2014; Sharf et al., 2006; Sharf and Ranaldi, 2006; Zellner and Ranaldi, 2010) we proposed that VTA DA cells receive the two critical signals for this type of neural plasticity to occur. We proposed that the US signal may be ACh release in response to primary rewards and the (eventual) CS signal may be glutamate release in response to environmental stimuli. Thus, the VTA may constitute a region where at least some of the neural plasticity underlying reward-related associative learning may occur. We now turn to evidence in support of the proposition that (1) VTA ACh constitutes a US signal and is critical for reward-related learning, (2) VTA glutamate constitutes a signal about environmental stimuli and is critical for reward-related learning.

Figure 1:

A representation of our neurobiological model of reward-related associative learning. The model stipulates that through conditioning rewards-associated stimuli can acquire the ability to activate VTA DA neurons in a similar way as primary rewards. A. During conditioning: A weak signal from reward-associated stimuli is sent through glutamate stimulation of NMDA receptors on VTA DA neurons. Concurrently, a strong signal from primary reward is sent through acetylcholine stimulation of muscarinic (mACh) receptor to VTA dopamine neurons. With concurrent stimulation the mACh-induced depolarization of DA neurons dislodges magnesium (Mg²⁺) ions from NMDA receptors, allowing the influx of calcium (Ca²⁺) into VTA DA cells. Influx of Ca²⁺ initiates intracellular cascades involving CaMKII, which results in several neuroplastic changes that strengthen the CS-associated glutamate synapses. B. After conditioning: After the CaMKII-mediated neural changes CS-initiated glutamate release causes bigger post-synaptic responses, strong enough to activate VTA DA cells and elicit conditioned responses.

In evaluating this model we considered the following types of evidence. (1) A sensory stimulus that potentially could be associated with a reward should be associated with the release of glutamate in the VTA. (2) A reward stimulus, such as food, should cause the release of ACh in the VTA and depolarize VTA DA neurons. (3) Antagonism of ACh neurotransmission in the VTA should attenuate the rewarding effects of rewards. (4) Antagonism of the glutamate signal, specifically at NMDA receptors (see below), in the VTA during the acquisition of learning should inhibit or impair the acquisition by the reward-associated stimulus (CS) of the capacity to elicit conditioned responses and activate VTA DA cells. (5) Antagonism of the ACh signal, specifically at muscarinic receptors (see below), during the acquisition of learning should inhibit or impair the acquisition by the reward-associated stimulus (CS) of the capacity to elicit conditioned responses and activate VTA DA cells.

2. VTA ACh as the unconditioned stimulus

In a series of papers we indicated that ACh release in the VTA served as an US signal that, in conjunction with a novel stimulus signal, and as would be predicted in a Hebbian model of learning, would lead to the synaptic plasticity that might be expected to occur when a novel reward-associated stimulus acquires the capacity to activate VTA DA neurons and elicit conditioned responses. As we pointed out in our series of 2006 papers, the Hoebel group demonstrated that extracellular concentrations of ACh in the VTA increase during eating, drinking and self-stimulation (Rada et al., 2000). Stimulation of mACh receptors in the VTA enhances brain stimulation reward (BSR) while mACh receptor antagonism reduces it (Kofman and Yeomans, 1988; Yeomans et al., 1993, 1985). Furthermore, mACh receptor antagonists in the VTA have been shown to reduce eating and approach to food (Ikemoto and Panksepp, 1996; Rada et al., 2000). Further, intra-VTA infusions of carbachol (a non-selective ACh agonist) induce CPP (Yeomans et al., 1985) and support operant self-administration (Ikemoto and Wise, 2002). Thus, these early studies indicated that reward stimuli cause the release of ACh in the VTA and that blockade of this neurotransmission reduces the rewarding effects of these stimuli. These findings demonstrate that VTA ACh plays a role in primary reward.

A major source of ACh to the VTA comes from the pedunculopontine tegmentum (PPTg) and laterodorsal tegmentum (LDTg) hindbrain nuclei (Garzon et al., 1999; Henderson and Sherriff, 1991; Oakman et al., 1995; Steidl et al., 2017b). DA neurons receive direct cholinergic inputs (Garzon et al., 1999) and express nicotinic (nACh) receptors (Azam et al., 2002; Durand-de Cuttoli et al., 2018; Mackey et al., 2012) and muscarinic (mACh) receptors [(the latter is suggested by indirect evidence from electrophysiology (Gronier and Rasmussen, 1998), microdialysis (Miller and Blaha, 2005), functional assays (Steidl and Yeomans, 2009) and in situ hybridization midbrain DA studies (Vilaro et al., 1990; Weiner et al., 1990)]. Application of ACh or its agonists depolarizes VTA DA neurons (Calabresi et al., 1989; Durand-de Cuttoli et al., 2018), causes burst firing (Durand-de Cuttoli et al., 2018; Gronier and Rasmussen, 1998) and DA release in the NAc and prefrontal cortex (PFC) (Blaha et al., 1996; Miller and Blaha, 2005; Schilstrom et al., 1998; Westerink et al., 1998). In contrast, intra-VTA infusions of scopolamine (a mACh antagonist) or mecamylamine (a nACh antagonist) diminishes DA release in the NAc evoked by brief electrical stimulation of the LTDg neurons (Forster and Blaha, 2000; Lester et al., 2008). Likewise, lesions to the LTDg reduce DA release evoked by cholinergic stimulation of VTA DA neurons with the intra-VTA acetylcholinesterase inhibitor neostigmine (Blaha et al., 1996), suggesting that DA neuronal activity is controlled by the LTDg->VTA acetylcholine pathway. Optogenetic activation of PPTg->VTA and LDTg-VTA acetylcholine pathways modulate the firing rates of VTA DA cells (Dautan et al., 2016) as well as non-DA cells (Yau et al., 2016). Electrical stimulation of the LTDg or PPTg (Forster and Blaha, 2000; Steidl et al., 2011) as well as chemical stimulation of both mACh and nACh receptors in the VTA has been shown to excite mesolimbic DA neurons (Calabresi et al., 1989; Gronier and Rasmussen, 1998; Lacey et al., 1990) and to facilitate release of DA in the NAc (Blaha et al., 1996; Miller and Blaha, 2005; Nisell et al., 1994).

The ACh projections from the LTDg and PPTg to the VTA not only control DA cell activity but appear to transmit a reward-related signal. In line with these findings are studies demonstrating that neurotoxic lesions to the PPTg that destroy cholinergic cells cause a significant reduction in well-acquired self-administration of nicotine (Lanca et al., 2000) and non-selective excitotoxic lesions to the PPTg impair the acquisition of self-administration of nicotine (Alderson et al., 2006), suggesting that these cholinergic neurons, which provide ACh transmission to the VTA, play a significant role in drug reward-related learning. Recent studies have shown that optogenetic stimulation of the LTDg->VTA or PPTg->VTA cholinergic pathways produces real time place preference and conditioned place preference (Dautan et al., 2016; Steidl et al., 2017b; Xiao et al., 2016) again demonstrating a role of these ACh pathways to the VTA in producing rewarding effects.

2.1. VTA muscarinic receptors in reward-related learning

Thus, VTA ACh release, by acting as a primary reward signal, might constitute the US signal in a hypothetical associative reward-learning model in the VTA. If this were so, then blockade of this ACh signal in a reward-related learning procedure should prevent the acquisition of such learning. We tested this hypothesis in a series of studies. In 2006, the Ranaldi group was the first to demonstrate that blockade of mACh receptors in the VTA impairs the acquisition of operant responding reinforced by food (Sharf et al., 2006). We later demonstrated that blockade of VTA mACh receptors during the acquisition of conditioned approach learning impeded this type of associative learning (Galaj et al., 2017; Ranaldi et al., 2011). Also, very importantly, our group demonstrated that the strength of this type of reward-related associative learning is significantly correlated with the degree of VTA DA neuronal activation by the CS (Galaj and Ranaldi, 2018). When animals were treated with scopolamine prior to conditioning sessions we observed significantly fewer TH-labeled (i.e., DA) cells in the VTA that expressed cfos and significantly less conditioned approach responding during the conditioned approach test (Galaj and Ranaldi, 2018). Scopolamine treatments made prior to the conditioned approach test did not affect responding. Altogether these results suggest that the degree to which a CS elicits conditioned approach depends at least partly on the degree to which the CS activates VTA DA cells and that the acquisition of these functions – the capacities by a CS to activate VTA DA cells and to elicit conditioned approach – requires mACh receptor stimulation during learning.

Overall, these findings suggest that mACh transmission in the VTA plays an important role in reward-related learning, and to some extent, in maintenance of US driven behaviors. More recent studies using optogenetics and Lox P technologies, which enable manipulation of selective populations of neurons, have demonstrated that ACh inputs from the LTDg to the VTA indeed play a role in reward-related learning. In ChAT-cre rats, stimulation of the selective LTDg->VTA or PPTg->VTA ACh pathways produces CPP for the stimulation-associated environment, suggesting that these ACh pathways play a role in reward-related learning (Xiao et al., 2016) and supporting the idea that ACh input to the VTA may serve a US function in our model. Likewise, optogenetic inhibition of the PPTg->VTA ACh pathway reduces time spent in the laser-paired compartment (Xiao et al., 2016) as well as the acquisition of conditioned approach for food (Yau et al., 2016), again indicating that the PPTg->VTA ACh pathway is critically involved in reward-related learning.

In regards to the LTDg ACh input to the VTA, Wise and colleagues have demonstrated that selective lesions to ACh neurons in the LDTg in rats with a history of cocaine self-administration delayed the initiation of responding for cocaine in the presence of predictive cues (Steidl et al., 2015). Interestingly, when primed, rats with LDTg lesions regained the typical pattern of cocaine self-administration, which might suggest that the LDTg->VTA ACh pathway may play a role in responsiveness to CSs but not operant responding reinforced with cocaine (Steidl et al., 2015). These data are inconsistent with previous findings demonstrating that the LTDg plays a role in responding to or for different drugs of abuse (Alderson et al., 2005; Dobbs and Cunningham, 2014; Forster et al., 2002; Nelson et al., 2007). The discrepancy in the results might derive from non-selective lesions or blockage of LTDg neurons. Electrical and neurotoxic interventions, which were commonly used to study the functional physiology of certain brain regions, do not offer spatiotemporal or cell-type specificity.

Note, our model stipulates that mACh neurotransmission in the VTA is necessary for acquisition of reward-related learning. In our model this activity is required for the neuroplasticity to occur that puts in place the mechanisms whereby future presentations of CSs can now activate this part of the reward system (i.e., mesolimbic DA neurons). An interesting question arises of whether or not, after the plasticity occurs, mACh receptor stimulation remains necessary for the performance of the already learned response. As stated above, we find no evidence of a role of VTA mACh receptor stimulation in the performance of already acquired conditioned reinforcement (Sharf and Ranaldi, 2006), operant (Sharf et al., 2005) or conditioned approach (Galaj et al., 2017) responses or on cfos activation in VTA DA neurons (Ranaldi et al., 2011), indicating that blockade of mACh receptor stimulation, in and of itself, is not sufficient to block the performance of already learned responses. However, others reported that intra-VTA infusions of atropine (a mACh antagonist) or mecamylanine blocked the expression of morphine CPP (Rezayof et al., 2007) and intra-VTA infusions of scopolamine or mecamylamine reduced cocaine seeking in rats (Nunes et al., 2019; Solecki et al., 2013). One possibility for the discrepancies is the use of higher doses in other studies compared to ours. Another is that blockade of ACh receptor stimulation reduces tonal activation of VTA DA neurons possibly reducing their baseline activity and making it more difficult for CSs to activate these cells. This would be more likely with larger than lower ACh antagonist doses. Certainly, future research should work out these discrepancies.

2.2. VTA nicotinic receptors in reward-related learning

The role of VTA nACh receptors in reward related learning is not well established as the literature provides conflicting results. For example, our group found no evidence of the role of VTA nACh receptor stimulation in the acquisition of food rewarded operant responding (Sharf et al., 2006). We are not aware of other experiments aimed at testing the role of VTA nACh stimulation in the acquisition of rewarding properties by neutral stimuli. However, there are several studies that have looked at the role of these receptors in the performance of conditioned responses. Thus, nACh receptors have been implicated in the performance of alcohol conditioned reinforcement (Löf et al., 2007), food conditioned reinforcement (Wickham et al., 2015), cue-induced cocaine seeking (Solecki et al., 2013) (You et al., 1998), sucrose seeking (Addy et al., 2015) and expression of cocaine CPP (Shinohara et al., 2014).

3. VTA glutamate transmits information about environmental stimuli

Our neurobiological model of reward-related associative learning stipulates conjoint activity of two inputs to the VTA DA neurons; one is a US and the other a (eventual) CS (from the reward-paired stimulus) (see Figure 1). Above we discuss how ACh in the VTA may constitute the critical US signal. Here we discuss how glutamate in the VTA may constitute the critical CS signal. There is evidence that VTA glutamate transmits information about environmental stimuli. Major inputs of VTA glutamate come from the PPTg (Charara et al., 1996; Floresco et al., 2003; Yau et al., 2016; Yoo et al., 2017), LTDg (Forster and Blaha, 2000; Lodge and Grace, 2006; Omelchenko and Sesack, 2005), lateral habenula (Brinschwitz et al., 2010; Geisler et al., 2007), bed nucleus of stria terminalis (BNST), superior colliculus (Coizet et al., 2003; Geisler et al., 2007) and PFC (Carr and Sesack, 2000; Gariano and Groves, 1998; Murase et al., 1993; Quiroz et al., 2016), (Georges and Aston-Jones, 2002, 2001; Jalabert et al., 2009) [for a comprehensive review of the functional role of VTA glutamate afferents see (Geisler and Wise, 2008)].

The PPTg is increasingly thought to mediate CS-associated signals that control VTA DA neurons. PPTg neurons respond to independent visual and auditory stimuli (Pan and Hyland, 2005) as well as those associated with rewards (Okada et al., 2009; Pan and Hyland, 2005). Projections from the LTDg are also involved in controlling the activity of VTA DA cells. Chemical, electrical or optogenetic stimulation of LTDg neurons results in glutamate release in the VTA (Nelson et al., 2007), burst firing of DA cells (Lodge and Grace, 2006) and DA release in the NAc (Forster and Blaha, 2000; Steidl et al., 2017a). Another source of VTA glutamate is the PFC (Carr and Sesack, 2000; Geisler et al., 2007; Quiroz et al., 2016; Sesack and Pickel, 1992; Vertes, 2004). Cortical glutamate neurons form synapses on VTA DA and GABA neurons (Carr and Sesack, 2000). A number of studies reported that electrical, chemical or optogenetic stimulation of different cortical areas leads to burst firing of VTA DA cells (Gariano and Groves, 1998; Murase et al., 1993; Tong et al., 1996).

It should be noted that the PFC provides relatively weak glutamate innervation to the VTA (Carr and Sesack, 2000; Sesack and Pickel, 1992). The VTA glutamate afferents express vesicular glutamate transporters (vGluT 1, 2, or 3) (Geisler et al., 2007). Those expressing the VGluT2 biomarker come from subcortical regions (e.g., PPTg, LTDg, lateral habenula, lateral hypothalamus) and those with vGluT3 primarily from the dorsal and median raphe (Geisler et al., 2007). However, cortical glutamate afferents innervating the VTA express mainly the VGluT1 biomarker, seen in scarcity in other VTA glutamate afferents (Geisler et al., 2007). Interestingly, the VTA has been reported to contain about five times more vGluT2 than vGluT1-expressing afferents, which suggests that the VTA is either weakly innervated by cortical glutamate afferents due to their arborization along the way (Vázquez-Borsetti et al., 2011; Vertes, 2004) or that not all of their RNA (situated in the nuclei of prefrontal cortical neurons) is transcribed into protein in their VTA terminals (Geisler and Wise, 2008). Perhaps, the cortical glutamate mRNA is transcribed and proteins are synthesized on demand or when reward-related learning neuroplasticity occurs. Clearly, more research is needed to fully understand the role of PFC-VTA connectivity in reward-related learning.

Thus, there is evidence that VTA DA cells receive glutamate signals about environmental stimuli. VTA neurons respond to novel stimuli (Dommett et al., 2005; Kiyatkin and Stein, 1996; Morrens et al., 2020; Pan and Hyland, 2005) and show less responsiveness to familiar non-reinforced stimuli over time (Ljungberg et al., 1992; Morrens et al., 2020; Schultz, 1998). In general, these neurons respond to salient events, including appetitive, high intensity or novel stimuli (Horvitz, 2000; Schultz, 1998)]. However, familiar stimuli can become salient conditioned stimuli during new conditioning (e.g., when paired with optogenetic activation of VTA DA neurons) and gain the capacity to activate VTA DA neurons to facilitate reward-related learning (Morrens et al., 2020; Saunders et al., 2018).

Exposure to novel environmental stimuli has been shown to elicit investigatory behavior in rats and increase release of DA in the NAc that can be blocked by intra-VTA infusions of kynurenic acid (a non-selective glutamate receptor antagonist), suggesting that phasic elevations in NAc DA evoked by exposure to novel stimuli depends on glutamatergic transmission in the VTA (Legault and Wise, 2001). One source of environmental stimulus-related signals is likely the PPTg. Electrophysiological studies show that PPTg neurons, some of which carry excitatory input to the VTA, respond to visual and auditory stimuli earlier than VTA DA neurons (Pan and Hyland, 2005), suggesting that these neurons might be the first recipients of information about environmental stimuli on its way to VTA DA cells. This is in line with studies demonstrating that lesions to the PPTg impair CS-US associative learning (Inglis et al., 2000), although it is possible that this learning blockade arises from disrupting US signals instead. In addition, the superior colliculus (a region involved in processing visual information), which provides glutamate input to the VTA, might be another source of environmental stimuli-related signals. Midbrain DA neurons receive glutamate afferents from the superior colliculus (Coizet et al., 2003; Geisler et al., 2007) and respond to the presentation of light (Dommett et al., 2005).

In our model these relatively weak glutamate signals act in conjunction with strong depolarizing ACh signals on VTA DA neurons to initiate the neural changes that transform these previously weak synaptic events into strong synaptic events that can themselves subsequently activate VTA DA cells and initiate conditioned responses similar to those of the primary reward. We have proposed that glutamate neurotransmission in the VTA plays two critical roles in reward-related associative learning. The first role is as an essential component of the acquisition of learning and involves stimulation of VTA NMDA receptors, which in conjunction with ACh depolarization of DA cells, results in intracellular neurochemical cascades that initiate the neural plasticity (re: formation of association). The second role is in the performance of the learned response. Here, glutamate release in the VTA occurs in response to presentations of CSs and stimulates post-learning potentiated synapses causing the conditioned activation of VTA DA neurons and its accompanying behavioral responses, which are now conditioned responses.

3.1. Stimulation of NMDA receptors in the VTA is necessary for the acquisition of reward-related learning

As we’ve laid out above, reward-associated stimuli have the capacity to activate VTA DA cells and we argue that this feature contributes to their capacity to function as CSs. The question we have been addressing is how do CSs acquire this capacity. We propose this happens with concurrent synaptic activity of VTA DA cells; one synapse strong enough to depolarize and activate the DA neurons and the other synapse at least active. For this model to be correct at least two other characteristics must exist. One is that these concurrent signals should be representative of the CS and US. We have presented evidence that mACh stimulation in the VTA can both depolarize VTA DA cells and function behaviorally as a US. We also present evidence that the VTA receives glutamatergic signaling from environmental stimuli, stimuli that could be associated with rewards and could potentially become CSs. The other necessary characteristic is that there be neurophysiological mechanisms in place that could produce the neuroplasticity that would result in a weak neutral stimulus signal becoming a strong CS signal. It has long been demonstrated that the NMDA receptor plays a necessary role in neuroplasticity (Collingridge et al., 2010; Jeziorski et al., 1994; Malenka and Bear, 2004; Nicoll, 2017; Wolf et al., 1994; Zweifel et al., 2008). VTA DA neurons are rich in NMDA receptors (James et al., 2015; Wang et al., 2011; Zweifel et al., 2008). This led us to hypothesize that stimulation of VTA NMDA receptors could play the critical role of initiating some of the neuroplasticity that would occur in reward-related learning.

Our model stipulates that the CS gains the capacity to activate VTA DA cells when concurrent stimulation of VTA NMDA receptors and strong depolarization of VTA DA cells, by mACh receptors in the VTA, occurs during the acquisition of reward-related behaviors (Ranaldi, 2014; Sharf et al., 2006; Sharf and Ranaldi, 2006; Zellner and Ranaldi, 2010). This coincident activity leads to the strengthening of CS-associated glutamate synapses and the control by CSs of VTA DA neurons that guide reward-directed behavior (Ranaldi, 2014; Zellner et al., 2009; Zellner and Ranaldi, 2010). The degree to which the CS can activate VTA DA neurons and elicits conditioned responses depends on the strength of associative reward-related learning that requires concurrent stimulation of NMDA and mACh receptors (Galaj and Ranaldi, 2018)

In a series of experiments we tested this model and demonstrated that blockade of VTA NMDA receptors during the acquisition of reward-related learning impedes conditioned approach learning (Ranaldi et al., 2011) or operant learning (Zellner et al., 2009) in rats. When animals were treated with MK-801 (a NMDA receptor antagonist) prior to conditioning sessions we observed significantly fewer TH-labeled (i.e., DA) cells in the VTA that expressed cfos and significantly less conditioned approach in response to a cocaine-associated CS (Galaj and Ranaldi, 2018). Similar pharmacological manipulations during the performance of already learned conditioned approach (Galaj and Ranaldi, 2018) or operant responding (Zellner et al., 2009) produce no effect on behavior or on cfos activation (Ranaldi et al., 2011). Similarly, intra-VTA infusions of NMDA receptor antagonists fail to reduce learned conditioned approach (Ranaldi et al., 2011) and operant responding (Zellner et al., 2009). Thus, blockade of NMDA receptors is not sufficient to block already learned conditioned responses indicating that, after associative learning has occurred, VTA NMDA receptor stimulation by itself plays a limited role in CS-signaling.

Hindbrain glutamate projections to the VTA also are strongly implicated in reward-related learning. Selective optogenetic inhibition of the PPTg->VTA glutamate pathway blocks the acquisition of conditioned approach (Yau et al., 2016), pointing to a critical role of glutamate input to the VTA in reward-related learning. In support of these findings are previous reports that lesions to PPTg disrupt the acquisition of stimulus-reward (conditioned approach) and conditioned reinforcement learning (Inglis et al., 2000) and sucrose CPP (Alderson et al., 2001). Caution must be applied in the interpretation of these experiments, however, as it is possible that these lesions destroyed non-glutamatergic pathways or pathways not projecting to the VTA which could have been responsible for the effect.

As animals learn CS-US (cue-reward) associations, they increase responding to CS presentations. During Pavlovian conditioning, transgenic mice lacking NMDA receptors exclusively on DA neurons show similar performance during CS-US presentations as their control littermates (James et al., 2015; Parker et al., 2011, 2010); however, they show attenuated phasic DA signaling (Parker et al., 2010), suggesting weaker CS-US associations and subsequent capacity to activate VTA DA neurons in the absence of NMDA receptors on DA neurons. Importantly, mice with selective deletions of NMDA receptors from DA neurons show impaired conditioned responding to CS presentations (Cieślak and Rodriguez Parkitna, 2019), suggesting the absence of NMDA receptors from DA neurons impairs conditioned approach learning in a way similar to pharmacological blockade of VTA NMDA receptors (Kest et al. 2012; Ranaldi et al. 2011). In addition, deletion of NMDA receptors from DA neurons has been shown to impair the acquisition of instrumental learning (James et al. 2015; Jastrzebska et al. 2016) and conditioned reinforcement (responses reinforced with CS presentations) (Cieślak and Rodriguez Parkitna, 2019) despite no change in motivation for food reward (Jastrzebska et al. 2016), suggesting that the absence of DA NMDA receptors impedes (1) the acquisition of reward-related learning and or (2) the ability of a CS to elicit or reinforce reward-driven behavior.

Our model stipulates that concurrent stimulation of mACh and NMDA receptors in the VTA leads to depolarization of VTA DA cells, resulting in dislodging the magnesium (Mg²⁺) ion from NMDA receptor, which in turn allows for the NMDA-induced influx of calcium (Ca²⁺) into VTA DA cells (see Figure 1). Influx of Ca²⁺ initiates intracellular cascades involving CaMKII, which results in several changes that strengthen the CS-associated glutamate signal. Thus, our model leads to the prediction that NMDA/mACh-initiated intracellular cascades are necessary for reward-related associative learning to occur. We recently tested this hypothesis and found that intra-VTA infusions of KN-93, a CaMKII inhibitor, impaired the acquisition, but not the post-learning performance, of conditioned approach and this effect was most pronounced with microinjections made in the middle and posterior portions of the VTA, suggesting that activation of CaMKII in these areas of the VTA is required for strengthening the CS-associated signal (Nisanov et al., 2020).

3.2. CS-associated enhanced glutamate release in the VTA as a CS signal

There are a number of studies that demonstrate a role of glutamate in the VTA in the expression of conditioned responses. We have shown that concurrent antagonism of AMPA or NMDA receptors is required to block the performance of an already acquired conditioned approach response (Hachimine et al., 2016). Intra-VTA microinjections of kynurenic acid (a non-selective ionotropic glutamate receptor antagonist) reduced conditioned approach whereas MCPQ (a metabotropic glutamate receptor antagonist) and AP-5 (a NMDA antagonist) did not. Interestingly, NBQX (an AMPA antagonist) produced small and non-significant reductions in conditioned approach in response to the CS, even at high doses (Hachimine et al., 2016). However, the combination of intra-VTA AP-5 and NBQX significantly reduced the performance of conditioned approach (Hachimine et al., 2016). This suggests that at least a component of AMPA receptor postsynaptic effects require NMDA receptor stimulation, an idea that is supported by other work (Lopez et al., 2015; Shi et al., 1999). If this is so, then it means that CS effects may be mediated by AMPA receptor stimulation, as LTP models indicate (Bredt and Nicoll, 2003; Malenka, 2003; Nicoll, 2003; Shepherd and Huganir, 2007), but also that AMPA effects are enabled by NMDA receptor stimulation (see Figure 1). These speculations will need to be tested. What we have observed is that the performance of a learned conditioned approach response requires either NMDA or AMPA receptor stimulation in the VTA.

Others also have shown that VTA glutamate release is a source of CS-associated signals (Yau et al., 2016; You et al., 2007). VTA glutamate levels are dramatically elevated when animals, experienced with drug self-administration, are placed in the drug-taking context prior to drug availability (You et al., 2007). Also, activity of PPTg->VTA glutamate neurons persists between the onset of cue presentation and reward delivery and the level of their activity correlates with the magnitude of the expected reward (Okada et al., 2009). Inactivation of the PPTg suppresses the conditioned responses of VTA DA cells to previously paired reward-associated stimuli (Pan and Hyland, 2005), a finding consistent with the idea that the PPTg->VTA, presumably glutamate, neuronal projections, carry CS-associated signals that mediate conditioned responses of VTA DA neurons to CSs.

These studies indicate that glutamate afferents to the VTA, which form synapses on DA cells and control their activity, can relay information about CSs. This strongly suggests that the brain regions of origin of these VTA glutamate afferents undergo their own neural changes whereby CSs gain the capacity to control those afferents to the VTA. Another possibility is that during the acquisition of reward-relating learning, there may occur VTA local signaling inducing plasticity in presynaptic glutamate terminals resulting in enhanced neurotransmitter release. Such mechanisms have been demonstrated as part of LTP (Nicoll, 2017). Thus, it seems that CSs come to control VTA DA cells through enhanced glutamate control of these cells. In one case this enhanced glutamate control comes about through VTA NMDA- and mACh-dependent activity resulting in likely local glutamate-related post-synaptic enhancements and in the other case through neural plasticity in one or more of the sites of origin of VTA glutamate afferents or, again through local pre-synaptic enhancements, which results in CS-related enhanced glutamate release in the VTA.

Conclusions

Our neurobiological model of reward-related learning is predicated on the stipulations that (1) the acquisition of reward-related learning occurs when reward-related stimuli (i.e., eventual CSs) acquire the ability to activate VTA DA neurons similarly to how natural rewards and drugs of abuse do and (2) the acquisition of this capacity to activate mesolimbic DA occurs with the concurrent stimulation of mACh and NMDA receptors in the VTA.

We hypothesize that VTA neurons receive mACh receptor signals representing USs and initially weak glutamate signals representing environmental stimuli some of which occur in close temporal or spatial proximity to primary rewards (i.e., USs). Initially, the glutamate signal is too weak to activate VTA DA neurons and cause a response. However, with concurrent stimulation of VTA mACh and NMDA receptors, the mACh receptor stimulation depolarizes VTA DA cells, removing the Mg²⁺ block from the NMDA receptor, allowing Ca²⁺ to enter DA cells. A rise in intracellular Ca²⁺ activates several enzyme protein kinases including CaMKII, leading to the strengthening of CS-related glutamate synapses and consequently of the CS signal. Thus, stimulation of mACh receptors in the VTA plays a critical role in strengthening CS-related glutamate synapses enabling reward-related learning to occur. Furthermore, either through local signaling or through neural events in regions other than the VTA, glutamate release in the VTA by reward-associated stimuli becomes enhanced, providing another route through which CS-related VTA enhanced glutamate signals activate DA neurons initiating conditioned responding. In the end, it seems that the psychological associations between rewards and the stimuli occurring in contiguity with them occur when neural associations between the neural substrates of the rewards and their associated stimuli are formed, enabling reward-associated stimuli to activate VTA DA neurons, a substrate that mediates at least some of the unconditioned effects of the primary rewards.

Highlights.

• Reward-associated stimuli can acquire the ability to activate dopamine cells

• VTA acetylcholine constitutes a primary reward signal

• VTA glutamate constitutes a conditioned stimulus signal

• This article bridges neuroscience with psychology of reward learning