Reinforcement signaling in Drosophila; dopamine does it all after all

Abstract

Reinforcement systems are believed to drive synaptic plasticity within neural circuits that store memories. Recent evidence from the fruit fly suggests that anatomically distinct dopaminergic neurons ultimately provide the key instructive signals for both appetitive and aversive learning. This dual role for dopamine overturns the previous model that octopamine signaled reward and dopamine punishment. More importantly, this anatomically segregated double role for dopamine in reward and aversion mirrors that emerging in mammals. Therefore, an antagonistic organization of distinct reinforcing dopaminegic neurons is a conserved feature of brains. It now seems crucial to understand how the dopaminergic neurons are controlled and what the released dopamine does to the underlying circuits to convey opposite valence.

Introduction

Learning provides animals with the ability to adapt their behaviour according to past and present circumstance. The presence of new and unexpected events in the environment, as well as the absence of predicted events are believed to be signaled in the brain by modulatory dopamine (DA) releasing neurons using a popular reinforcement learning mechanism [1,2]. In doing so these DA neurons provide an ongoing update of the environment and tune synaptic activity in the underlying neural circuitry so that current and future behaviour is optimal. Understanding whether such a reinforcement learning model represents a genuine and conserved mechanism of behavioural control in the nervous system, and if so how it operates, is of great interest and will require analyses across phyla. The enhanced cellular resolution offered by studying the genetically tractable and numerically less complex fruit fly brain provides considerable potential to understand DA neuron functions at the molecular, cellular, physiological and small neural network level.

Dopamine signals aversive reinforcement

Drosophila DA was previously thought to signal aversion and octopamine (OA), the invertebrate analog of norepinephrine, attraction [3,4]. Early studies disrupting DA neuron output during appetitive and aversive olfactory conditioning concluded that DA was exclusively required for aversive reinforcement [3]. Subsequently, using optogenetics, a subset of 12 DA neurons was identified in the PPL1 cluster (Figure 1a) whose activation paired with odour presentation could implant aversive memories [5]. Similar studies using thermogenetics to trigger other subsets of DA neurons in PAM (Figure 1b) and the PPL1 group produced effective negative reinforcement [6,7•]. In addition blocking these neurons impaired the formation of electric-shock reinforced aversive memory. It is therefore clear that certain DA neurons in PPL1 and PAM can provide negative valence if engaged during odour presentation [5-7•] (Figure 2a).

Figure 1. Innervation of the mushroom body by DA neurons.

(a) The protocerebral posterior lateral (PPL) 1 cluster. The Butcher’s cut illustration is edited from [19]. The image shows one neuron projecting to the tip of the α lobe (red), one to the tip of the α′ tip (orange), one to the upper stalk of the vertical lobes (MB-V1 neuron, yellow), one to the lower stalk and junction region (MB-MV1 neuron, blue) and one to the heel and distal peduncle (MB-MP1 neuron, green). All neurons shown have a projection to a similar zone on the contralateral MB (dotted lines with arrowhead). The number of neurons in each class has not been determined but there appears to be at least two MB-MP neurons per PPL1 cluster [6,18].

(b) The PPL2ab and protocerebral anterior medial (PAM) clusters. The illustration summarizes data described in [6,12••,15••,49]. At least two neurons from the PPL2ab cluster innervate the ipsilateral MB calyx (brown and blue). PAM DA neurons project to discrete zones of the horizontal β, β′ and γ MB lobes (marked with orange dotted lines for γ). The aversive MB-M3 neurons ramify on the tip of the β lobe (red). Several of the PAM DA neurons also have a projection to the contralateral MB (dotted line with arrowhead). Cell body position is not stereotyped and diagrams are not intended to be anatomically accurate.

Figure 2. Models for the involvement of DA neurons in aversive and appetitive reinforcement.

Both models are updated from those in [19]. (a) DA neurons representing aversive reinforcement. Studies [5-7] suggest that neurons in PPL1 convey negative reinforcement whereas live-imaging data [49] corroborate that DA neurons innervating the lower stalk and junction are strongly activated by shock. In this illustration a selection of DA neurons innervating these MB zones and the MB-M3 neurons on the tip of the β lobe are activated (red).

(b) DA neurons representing appetitive reinforcement. DA neurons in PAM innervate many discrete zones in the β, β′ and γ lobes [7,12••]. Ca²⁺ imaging of sugar-evoked activity suggests only some zones receive appetitive reinforcement[7]. Those DA neurons innervating the β′ and γ lobe tips are modulated by OA through OAMB [12] (yellow outline). OA-dependent reinforcement also requires OCTβ2R in the aversive MB-MP1 neurons (yellow). Some of the PAM DA neurons on the β′ and γ lobe tips are also required to mediate the OA-independent reinforcing effects of nutrient value.

Octopamine signals appetitive reinforcement from sweet taste

A role for OA in reward came from seminal studies in the honeybee that showed that electrical stimulation of a single octopaminergic neuron, or injection of OA into the honeybee antennal lobes or MB calyx, could replace the presentation of a sucrose reward in olfactory conditioning of the proboscis extension reflex [8,9]. Consistent with the bee studies, Tbh mutant flies that lack octopamine cannot form appetitive memory [3]. Additionally, optogenetic activation of OA neurons formed appetitive olfactory memories in Drosophila larvae [10].

Recent work utilizing a variety of natural sugars as reinforcers in learning revealed that sweet taste and nutrient value represent parallel appetitive reinforcement pathways in fruit flies [11•]. Acutely blocking output from OA neurons disrupted learning with arabinose, a sugar that tastes sweet but that the flies cannot metabolize [12••]. However, blocking OA neurons lacked consequence when flies were conditioned with sweet and nutritious sucrose. It therefore appears that OA only provides the transient reinforcing properties of the sweet taste of sugar and that nutrient value provides additional reinforcement and bypasses the requirement for OA. Consistent with this model, only short-lived memories are formed if adult flies are ‘artificially conditioned’ by pairing odour presentation with OA neuron activation [12••]. So how does OA act and what neurons mediate the reinforcing effects of nutrient value?

Dopamine signals appetitive reinforcement of sweet taste and nutrient value

Analysis of dumb flies mutant for the DopR dopamine receptor provided the first indication that dopamine was also involved in fly reward learning [13]. Mutant dumb flies are defective in both aversive and appetitive learning and the defect can be rescued by restoring DopR expression to the mushroom bodies (MB) [13,14•]. Strikingly, appetitive memory could not be implanted with OA neuron stimulation in dumb mutant flies suggesting that DA signaling is functionally downstream of OA in appetitive memory processes [12••].

The prior conclusion that DA was only required for aversive learning was drawn from studies using a TH-GAL4 line that does not express in all of the fly’s DA neurons [3,5]. Remarkably, we now know that among the DA neurons that are omitted by TH-GAL4 are those in the PAM cluster that signal rewarding reinforcement [12••,15••] (Figure 2b).

Two studies independently identified GAL4 driver lines that express in large numbers of PAM DA neurons that are not included in the TH-GAL4 cohort [12••,15••]. Pairing odour exposure with activation of these PAM DA neurons formed robust appetitive memory, even in flies lacking OA. Importantly, blocking PAM DA neurons disrupted appetitive learning with sweet only sugar [12••] and persistent appetitive memory formation with nutritious sugar [12••,15••]. Therefore PAM DA neurons lie downstream of OA signals and appear to represent both the short-term and long-term reinforcing effects of nutritious sugar [12••].

Further analysis revealed that the short-term reinforcing effects of OA go through the PAM-DA neurons (Figure 2b). Memory cannot be formed with OA neuron activation in flies that lack the α-adrenergic-like OA receptor OAMB and knocking the Oamb gene down specifically in PAM-DA neurons impairs conditioning with the sweet-only sugar arabinose, but not with nutritious sucrose [12••]. Lastly, the OAMB receptor couples to the release of intracellular Ca²⁺ [16,17] and OA application to the brain evoked a robust increase in cytosolic Ca²⁺ in PAM-DA neurons [12••] consistent with a direct action of OA through OAMB.

Zonal organization of reinforcing DA neuron input to the MB

Relative valence is therefore conveyed to different regions along the processes of MB neurons by anatomically distinct DA neurons. The two types of PPL1 cluster DA neuron that can provide aversive reinforcement innervate the vertical MB lobe and heel regions [5-7•] (Figure 2a). In contrast, the larger group of 40-80 PAM cluster neurons that can provide appetitive reinforcement exclusively innervate the horizontal MB lobes [12••,15••] (Figure 2b). However, a model that segregates positive and negative valence between the MB neuron collaterals is too simple because one subset of PAM-DA neurons that can convey aversive reinforcement innervates the horizontal β-tip region [6] (Figure 2a). Therefore, aversive DA neurons are distributed between the PPL1 and PAM clusters whereas the PAM cluster contains DA neurons with opposing valence. Moreover, recent experiments suggest that OA-directed appetitive reinforcement also requires simultaneous modulation of the MB-MP1 neurons through a β-adrenergic like OA receptor OCT2βR [12••]. The MB-MP1 neurons can provide aversive value if artificially engaged [6] and have been shown to be hunger-modulated by Neuropeptide F (the fly analog of Neuropeptide Y, NPY) to control appetitive motivation [18]. It therefore seems likely that value is signaled in the MB through a complex concerted action of the different types of DA neurons [19].

What input pathways drive DA neurons?

The activity of the appropriate classes of fly DA neuron must be coordinated by input pathways. Therefore, delineating relevant afferent neurons should reveal the nuances of the functional organization of the fly DA system.

Work suggests that the MV1 and MP1 PPL1 neurons and the PAM-M3 neurons are all required to convey the reinforcing effects of electric shock [7•]. These neurons have dendrites in a similar brain area and could plausibly be driven by the same afferent neurons. However, we do not know the peripheral input pathways, or interneurons, that relay shock information to these DA neurons. Nor do we know whether the non-physiological shock stimulus will be represented in a specific pathway. Ultimately, it will be important to determine whether DA neurons signaling aversive value are engaged together by other more ecologically relevant aversive stimuli. Bitter tastants [20-22] and post-ingestive reinforcing effects of toxin [23•] may represent a good way forward.

OA neurons provide a critical input to DA neurons for appetitive reinforcement with OA simultaneously modulating positive PAM and negative MB-MP1 DA neurons through α- and β-adrenergic like receptors respectively [12••]. We do not know if OA modulates the activity of additional DA neurons and whether other appetitive reinforcers activate the same or different OA and DA neurons. Likewise, the input representing nutrient value to PAM-DA neurons does not involve OA and remains to be identified.

What does DA do to MB neuron synapses?

The zonal organization of reinforcing DA neurons on the MB (Figure 2) suggests that the underlying MB synapses are the substrate of memory-relevant synaptic plasticity. It is therefore predicted that the relevant output neurons will have processes within the MB in tight opposition to those of reinforcing DA neurons, maintaining the zonal architecture [5]. Anatomical work is consistent with this model [24]. However only the MB-V2 output are known to be important for behaviour [25••]. The <70 MB-V2 neurons have dendrites that densely innervate the MB α-stalk in the general vicinity of the MB-V1 DA neurons, that do not have a strong reinforcing role [7•,25••]. Nevertheless, MB-V2 output is required for retrieval of aversive memory [25••] consistent with memory being represented in the synapses between MB and MB-V2 neurons.

Identifying functionally relevant MB-V2 output neurons allows one to test the consequence of learning at MB neuron output synapses. Aversive learning decreased the odor-driven activation of MB-V2 neurons [25••]. Although it is perhaps surprising that learning depresses synaptic weight between the MB Kenyon cell (KC) and the MB-V2 output neurons, a comparable decrease has been describe at the KC-β-lobe output synapse in the locust brain following OA application [26••].

As their name suggests, locust β-lobe output neurons (bLN) are a group of 10-100 neurons that pool KC outputs in the MB beta lobe. Electrophysiological analyses revealed that individual bLNs are broadly tuned to odors and that the KC-bLN synapses exhibit Hebbian spike-timing dependent plasticity (STDP) [26••]. Importantly, OA application to the brain specifically depressed STDP marked, or odour-evoked, synapses. Moreover, even OA application that was delayed for up to one second after the STDP window caused synaptic depression. Therefore, STDP can temporarily ‘tag’ synapses making them susceptible to subsequent modulation by OA. Such an STDP mechanism that tags odour-activated KC-output synapses allows a temporally delayed and broadly targeted reinforcement signal to gain odour and synaptic specificity [27].

Given the recent evidence that OA works through DA [12••] and that DA ultimately signals aversion and reward, it seems possible that dopamine is responsible for both the aversive learning effect in fruit flies and the OA plasticity in the locust.

How informative is a comparison to the mammalian DA system?

DA has been classically associated with pleasure and addiction/motivation in mammals but it is now clear that DA also mediates aversion. Subsets of mammalian DA neurons respond to rewarding, rewarding and aversive stimuli or aversive stimuli [28-31]. In addition DA neurons are required for aversive learning [31,32].

Aversive Ventral Tegmental Area (VTA) DA neurons project to medial prefrontal cortex and onto GABAergic neurons in the rostromedial tegmental area. In contrast rewarding VTA DA neurons project to the nucleus accumbens [33,34••]. Some mammalian rewarding DA neurons are inhibited by aversive cues [35] by neurons in the lateral habenula that code a negative prediction error [36]. Furthermore, hypothalamic NPY/AgRP neurons in the mouse control motivation and rewarding DA neurons [37]. It is therefore clear that the fly and mammalian DA systems are similarly organized and so it seems likely that some lessons learned in flies will be relevant to DA reinforcement systems in mammals and vice versa.

Mammalian DA neuron inputs are largely glutamatergic and GABAergic. Aversive VTA DA neurons receive glutamatergic input from lateral habenula neurons. In contrast rewarding VTA DA neurons receive glutamatergic afferents from the laterodorsal tegmentum [33,34••,38]. Furthermore, GABAergic neurons in VTA are activated by aversive stimuli and encode reward expectancy [39••]. These VTA GABAergic neurons in turn inhibit the activity of rewarding DA neurons. So far such a detailed input is missing in the fly DA system.

Knowledge of input allows one to alter the activity of DA neurons. For example, NMDA glutamate receptor deletion from DA neurons in the mouse selectively impaired phasic burst firing without altering tonic activity [40]. Strikingly these mice had impaired reward and punishment learning while other DA-dependent behaviours were unaffected. Deleting a subunit of the GABA(A) receptor from DA neurons enhanced reward learning while leaving aversive learning intact suggesting an important role for DA neuron inhibition in appetitive reinforcement [41].

Work in mammals has revealed a complex interaction between STDP and DA [42]. DA receptor activation is often a prerequisite for STDP [43,44] and can set the conditions for plasticity. Most interestingly DA can widen the STDP window, decrease the number of spike pairings required to induce timing-dependent long-term potentiation (t-LTP), and can even sign reverse timing-dependent long-term depression (t-LTD) into t-LTP [45]. It could be that some MB output synapses are subject to OA/DA based repression whereas others are potentiated by OA/DA or DA alone. Earlier experiments suggested that only neurotransmission from the α′β′ subset of MB neurons is required during fly learning [46]. Uncoupling αβ and γ presynaptic release from postsynaptic neural activity lacked consequence [46-48] presenting a conundrum for a straightforward STDP model of fly learning in these neurons. It will therefore be critical to clarify the mechanisms of DA action and the synaptic rules of learning throughout the MB neuron ensemble.

Highlights.

• Dopamine represents appetitive and aversive value in the fruit fly brain.

• Dopamine neurons representing opposite value are anatomically segregated.

• Octopamine conveys appetitive reinforcement of sweet taste through dopamine neurons.

• A dual role for dopamine in appetitive and aversive value is conserved in mammals.