Serotonergic Modulation of Olfaction in Rodents and Insects

Abstract

Recent advances in genetic tools and optical imaging technology have allowed rodent and Drosophila researchers to explore the relationship between serotonergic modulation and olfactory processing at a mechanistic level previously unfeasible. Here, I review the basic organization of olfactory and serotonergic systems in both rodents and Drosophila and draw comparisons where similarities exist. I discuss circuit level models that explain many of serotonin’s effects on olfactory responses in the olfactory system’s inputs and outputs. Finally, I discuss models of integration within wide-field centrifugal neurons to emphasize the importance of studying serotonergic neurons directly to build more realistic models of olfactory and modulatory interactions.

Introduction

In this review, I will describe the interactions of the olfactory and serotonergic systems in both rodents and insects, with an emphasis on recent work in mice and Drosophila. Each species has its own unique advantages, and rodent and insect models both have a rich history in sensory neurobiology. The rodent anatomy closely resembles that of our own with homologous olfactory and serotonergic structures. Insects have both a reduction in neuronal number and uniquely identified neurons that can be targeted across individuals. The precise function of serotonin (5-HT) in olfaction remains elusive, though the modulator has been implicated in olfactory learning [1,2], promoting adult neurogenesis [3], and modifying ongoing sensory coding [4,5]. In this review, I will focus on the latter and discuss the effects of serotonin on different cell types within early olfactory processing. Additionally, I will discuss the need and challenge to better correlate serotonin release with olfactory signaling to understand the contexts under which such modulation might occur. The hope is to highlight similarities between rodents and insects in order to find common principles that likely generalize more broadly across taxa. Recent advances in imaging technology and genetic tools have provided new insight into the modulation of chemosensation that was not feasible only a few decades ago.

The Olfactory System and its Serotonergic Innervation

Olfaction begins with an odor molecule binding to an olfactory receptor (OR) expressed by an olfactory sensory neuron (OSN). Each OSN expresses a single OR type [6]. In mammals, OSN axons project across the cribriform plate and terminate in specialized neuropil called glomeruli within the olfactory bulb (OB). Each OSN expressing the same OR type converges onto the same glomerulus where they make synapses onto a variety of postsynaptic partners including principal neurons and local interneurons [6-10] (Figure 1A). The principal neurons of the rodent olfactory bulb are the mitral cells (MCs), which project to piriform cortex [11] and the amygdala [12] to drive learned and innate olfactory behaviors, respectively. In addition to principal neurons, the OB also possesses numerous types of local interneurons that mediate lateral excitation, pre- and postsynaptic inhibition, and intraglomerular inhibition [8,13-15]. Like most regions of the brain, the OB is subject to modulation by serotonin (5-HT). As there are no intrinsic serotonergic neurons in the OB, serotonin is instead supplied via centrifugal innervation from the raphe nuclei in the brainstem [16,17] (Figure 1B). Serotonergic fibers are densest in the glomerular layer, which is innervated by the median raphe nucleus (MRN) [16,18]. Fibers from the dorsal raphe (DRN) target the mitral and granule cell layers of the OB, as well as the piriform cortex and the amygdala [18].

Figure 1.

The organization of the olfactory and serotonergic systems in mice and flies. (A). A schematic of the layers and major cell types in the mouse olfactory bulb (OB). Olfactory input comes from OSN axons in the olfactory nerve layer (ONL). OSNs synapse onto mitral cell (MC, pink) dendrites in glomeruli in the glomerular layer (GL). The mitral cells project out of the OB and to third order olfactory regions such as the piriform cortex (pir) and the amygdala (am). Juxtaglomerular neurons such as short axon cells (SAC, blue), periglomerular cells (PG, yellow), and external tufted cells (Te, green) modify olfactory processing in the glomerular layer. SACs and PG are inhibitory and Te cells are excitatory. Granule cells (GC, gray) in the granule cell layer (GCL) send dendrites to the external plexiform layer (EPL) where they make dendrodendritic synapses with MC. Serotonin receptors that have been attributed to each cell type are listed next to that cell. Serotonin receptor classes that have only been localized to layers of the bulb are listed next to that layer on the right. The median raphe nucleus (MRN) provides serotonergic innervation for the GL (light red shaded layer), and the dorsal raphe nucleus (DRN) provides innervation for the mitral cell layer (MCL) and granule cell layer (GCL) (light shaded blue). The most densely innervated layer is the GL. (B). The DRN projects from the brainstem to three major olfactory areas, the OB, the piriform (pir), and the amygdala (Am). MRN projections to the OB also arise from the brainstem. PFC = prefrontal cortex, Cx = cortex, HPC = hippocampus, thal = thalamus. Adapted from Muzerelle et al. 2016 with permission. (C). A schematic of the Drosophila antennal lobe (AL). OSNs residing on the antennae each project to a single glomerulus in the AL. Each OSN expressing the same receptor (denoted as the same shade of gray) project to the same glomerulus where they make synapses onto projection neurons (PN, pink). PNs send axons out of the AL to the lateral horn (LH) and mushroom body (MB) where third order olfactory processing takes place. Inhibitory (blue) and excitatory (green) local interneurons also influence glomerular processing in flies. Only two cells (one from each hemisphere) provide serotonergic innervation for the AL. These are the CSDns. (D). A schematic of the Drosophila brain showing the location and projections of the major cell classes discussed in this review.

The early olfactory system in insects is organized in a manner highly analogous to the rodent OB, implying that lessons and principles learned in one species may generalize to others [19-23] (Figure 1C). As in mammals, most insect OSNs also express a single OR and project to a dedicated glomerulus in the antennal lobe (AL) [7,24,25]. Projection neurons (PNs) in the AL serve a similar role to the mammalian mitral cells and relay olfactory information to higher brain regions [26-28]. These regions include the mushroom bodies for processing learned olfactory behaviors and the lateral horn for mediating innate olfactory behaviors [29]. Additionally, the AL possesses a variety of excitatory and inhibitory interneurons that shape PN responses through intra- and interglomerular interactions [27,28,30-35]. In nearly all insects that undergo metamorphosis, including flies, only one pair of serotonergic neurons innervates the AL [36,37]. These cells, termed contralaterally-projecting, serotonin-immunoreactive deutocerebral neurons (CSDns) also provide all of the serotonergic innervation for the lateral horn [38]. The mushroom bodies of the fly, however, are innervated by multiple 5-HT cell types [36,39,40].

Serotonin and Olfactory Coding

The major effects of serotonin in the OB are to suppress OSN output [4] (Figure 2A) and to modulate mitral cell activity [41-44]. The suppression of OSN terminals is indirect and mediated by the activation of excitatory 5-HT_2C receptors on GABAergic short-axon cells (SACs) [4,42,43] (Figure 2B). Notably, no serotonin receptors are expressed by OSNs in the OB and raphe axons do not target their terminals for direct modulation [45]. As SACs presynaptically inhibit OSN terminals in the OB, elevation of their activity via 5-HT reduces OSN transmission [4]. External tufted cells (ETCs) are a major source of excitation for SACs, and ETC activity is also potentiated by 5-HT via 5-HT_2A receptors [42,46]. Thus, serotonergic inhibition of OSNs is indirect and mediated by both the direct and indirect recruitment of SACs. OSN inhibition may serve as a form of global gain control to decrease olfactory sensitivity [4,47]. Interestingly, OSN suppression is not glomerulus or odor specific [4]. If intense olfactory stimulation were to drive raphe activity in the OB, then this mechanism could prevent the saturation of mitral cell responses. Alternatively, suppression of OSN responses may be an important mechanism by which the brain could use serotonin to gate out sensory information that might otherwise compete for attention or other cognitive resources. Serotonin in the leech nerve cord serves a similar function by gating out tactile signals that compete with feeding behavior [48].

Figure 2.

Cellular mechanisms for 5-HT modulation in the OB and AL (A). Strong raphe stimulation results in long-term suppression of OSN output. Adapted from Petzold et al. 2009 with permission. (B). 5-HT acts on several cells in the OB. It directly excites ETC which in turn excite SACs. SACs are also directly excited by 5-HT. These two mechanisms serve to indirectly inhibit OSN axons and thus mitral cell activity. However, 5-HT can also excite MC directly or indirectly through ETC. Thus, the net effect of 5-HT on mitral cells can be either excitatory or inhibitory in a glomerulus specific fashion. (C). Exogenous application of 5-HT boosts mitral cell responses to olfactory nerve shock and suggests a net excitatory role for 5-HT in the OB. Note that the boosting is only seen in absolute terms and that the change from baseline is similar in saline and 5-HT. Adapted from Brill et al. 2016 with permission. (D). However, blocking 5-HT signaling from the raphe with methysergide (methy, 50 μm) boosts other mitral cell responses to current injection showing that 5-HT can have a net inhibitory effect on OB output as well. Compare number of gray and red responses below the unity line. Adapted from Kapoor et al. 2016 with permission. (E). In flies, 5-HT modulates PN output in a non-monotonic fashion where low 5-HT levels suppress output and high 5-HT levels boost PN output. Adapted from Zhang et al. 2016 with permission. (F). A schematic of the AL circuitry that can explain the non-monotonic nature of 5-HT modulation in insects. Inhibitory LNs (blue) possess both excitatory and inhibitory 5-HT receptors. GABAergic LNs typically inhibit OSN output but can also target post-synaptic PNs. (G). Excitatory 5-HT₇ and 5-HT₁ receptors have greater sensitivity to serotonin than the inhibitory receptors in Drosophila. Thus, low levels of 5-HT may preferentially boost LNs to suppress PN output, while high 5-HT levels may inhibit LN activity and boost PN output. Methysergide would remove excitation to LNs from resting 5-HT levels, thus dis-inhibiting OSN-PN odor responses. Adapted from Gasque et al. 2013 with permission. (H). Optogenetic stimulation of the Trh-Gal line (which does not label the CSDn) inhibits DA1-PN odor responses. Suppression of DA1-PN output requires sustained Trh-Gal4 activation. Adapted from Zhang et al. 2016 with permission.

The effect of 5-HT on mitral cell output is far more complicated. Mitral cells as a population express both excitatory 5-HT_2A [49] and inhibitory 5-HT₁ [41] receptors. Thus unsurprisingly, MCs can either be excited (Figure 2C) [42,44] or inhibited [41] by 5-HT. Application of the broad serotonin antagonist methysergide increases mitral cell excitability in response to the stimulation of raphe axons, thus suggesting an inhibitory role for 5-HT from endogenous sources [41] (Figure 2D). Additionally, some MCs are inhibited by 5-HT in an indirect, GABA-sensitive manner [44]. This suggests that the overall effect of 5-HT on MCs is bidirectional and likely dependent on both the direct actions on the MC as well network interactions.

What function might the bidirectional modulation of MC activity serve? One possibility is to preferentially allow some glomeruli to respond to odors despite the indirect inhibition of their cognate OSNs in the presence of 5-HT. This scenario could allow 5-HT-boosted MCs to maintain their dynamic range in spite of reduced sensory drive, and may also allow the animal to perceive a limited set of odors while blocking out interference from other odors. Alternatively, the modulation might serve to increase the discriminability between odors by increasing the difference in firing rates between MCs. As described above, the MC layer receives serotonergic innervation predominantly from the DRN, while the glomerular layer is densely innervated by the MRN. Thus, global gain control of olfactory input could occur independently of direct MC modulation. It is unclear under what conditions differential release of 5-HT into the mitral cell layer versus the glomerular layer might occur. While the DRN and MRN have extensive overlap in the regions from which they receive their inputs, some distinctions do exist [50,51]. For example, the DRN receives a greater percentage of its inputs from higher olfactory regions [50].

Many similarities exist between serotonergic modulation in the OB and the insect AL. Serotonin also inhibits insect OSN output in an indirect, GABA-dependent manner [5], and 5-HT differentially modulates PN activity [5,52]. Additionally, as in mice, both exogenous 5-HT and methysergide can boost Drosophila PN responses [5,53]. This makes sense in insects as PN odor responses are modulated in a non-monotonic fashion. High concentrations of serotonin boosts PN responses in flies and moths and low concentrations suppress odor responses [5,53,54]. Drugs like fluoxetine and methysergide both manipulate serotonin in the low concentration range and reveal its suppressive nature, whereas exogenous application of 5-HT demonstrates an excitatory function at higher concentrations (Figure 2E). This phenomenon is best explained by the expression of both excitatory and inhibitory 5-HT receptors within the GABAergic LN population [55] (Figure 2F). Because the excitatory 5-HT receptors are activated at lower serotonin concentrations [56] (Figure 2G), it is possible that LNs are first activated to suppress olfactory output, and subsequently inhibited through the activation of inhibitory 5-HT receptors at increasing concentrations. Indeed, strong activation of the CSDn does inhibit most AL neurons via 5-HT, including GABAergic interneurons [53]. The function of this non-monotonic modulation is not known, but it does add an extra layer of complexity in understanding the interaction between serotonin and olfaction.

As described above, reasonable receptor-based models exist to explain the net effects of 5-HT on rodent and insect early olfactory neurons. However, in both phylogenetic groups, these models are based on a small subset of the 5-HT receptors present in the OB and AL. In the OB, at least 9 types of 5-HTRs have been identified throughout the various layers [57-62] (Figure 1A). The function of the majority of these receptors in olfaction is unknown and virtually unexplored. Likewise, the Drosophila genome encodes 5 serotonin receptors [56,63-65] and all 5 receptor classes are broadly distributed throughout multiple cell types in the AL [55]. The mechanisms by which different classes of 5-HT receptors are recruited and how they each influence signal processing will be an important part of understanding serotonin's function in olfaction.

Endogenous Modulation and Multiple Transmitters

It is important to distinguish the effects of serotonergic modulation versus modulation by serotonergic neurons. Because raphe neurons and the CSDns both possess a fast neurotransmitter in addition to 5-HT [41,53,66-68], stimulating these neurons can modify neuronal activity in unpredictable manners. Rather than the effects of 5-HT alone, what may be most important is the balance of excitation and inhibition that results from the release of both transmitters when serotonergic neurons are excited. In mice, stimulation of raphe axons results in some MC odor responses being potentiated while others are suppressed [41,43]. The bidirectional modulation of MC odor responses to brief raphe stimulation does indeed increase the discriminability of odors across the OB in vivo [41]. Though it remains unclear precisely how much of the improvement in odor separation is attributable specifically to glutamate, serotonin, or their combined actions. One possibility is that discriminability improves with brief raphe stimulation predominantly via glutamate, while serotonergic modulation only occurs at higher activity levels. At those higher activity levels, raphe function could switch to regulating global gain control or the enhancement of select olfactory channels. In this model, the effects of glutamate and serotonin are inherently separable based on the activity level of the raphe nucleus. Indeed, long-term serotonergic modulation of OSN responses in mice requires fairly sustained raphe activation while glutamate release is seen even with brief stimulation [4,41].

Drosophila CSDns also release an excitatory co-transmitter along with 5-HT [53]. In flies this transmitter is acetylcholine (Ach), which signals predominantly through nicotinic receptors in the AL [69]. Optogenetic activation of the CSDns reliably elicits 5-HT mediated inhibitory currents and Ach-mediated excitatory currents in PNs and LNs [53]. As with mice, serotonergic modulation in flies also requires prolonged activity in serotonergic neurons [53] (Figure 2H), suggesting Ach and 5-HT effects might be dissociable. No studies to date have looked at the effects of direct CSDn stimulation on population coding in PNs, so it is unknown if CSDn activity might also ameliorate odor discrimination in the AL.

Activation of Serotonergic Neurons and the Release of 5-HT

Developing realistic and ethologically relevant models of 5-HT function in the OB and AL will ultimately require knowing under what conditions 5-HT is released into these structures. For example, if the function of 5-HT is for gain control to prevent MC saturation, we might expect activation of the raphe nucleus to correlate with activation of olfactory areas. Alternatively, a model where 5-HT blocks out competing olfactory input might require the raphe nucleus to be activated by brain regions unrelated to olfaction. Unfortunately, determining the conditions that specifically activate OB- or AL-projecting 5-HT neurons has proven difficult. Several studies in rodents have recorded from raphe neurons during olfactory tasks and have found their odor responses to be highly variable across cells [70,71]. Importantly however, it cannot be determined which of these neurons project to the OB.

While physiological evidence for the activation of OB-projecting raphe neurons is ideal, insight regarding 5-HT release into the OB can be gleaned from the anatomical inputs that innervate the raphe [50,51]. Both the DRN and MRN receive substantial input from the hypothalamus and 5-HT levels in the OB fluctuate with circadian rhythms [72]. Other prominent inputs to the raphe nuclei include midbrain structures, such as the striatum, and ventral pallidum. The raphe also receives more minor inputs from the amygdala and cortical regions including piriform cortex. However, similar to the electrophysiological experiments described above, such current anatomical studies have focused on the raphe nuclei as a whole and do not necessarily reveal the predominant inputs specifically to OB-projecting neurons. It is of course possible that the population average inputs to the raphe may not accurately reflect the relative presynaptic inputs to those raphe neurons that project to the OB.

The CSDns present a unique opportunity to ask specifically how serotonergic neurons that project to olfactory areas respond during olfactory stimulation. This is because they are the only serotonergic fibers to innervate the AL, and they can be targeted genetically [73,74] for whole-cell physiology [53]. Surprisingly, the CSDns in Drosophila are broadly inhibited by virtually all odors, and this inhibition arises from local interactions within the AL [53]. The inhibition scales with odor intensity. Major presynaptic partners of the CSDn in the AL are the GABAergic LNs, which are also recruited proportionally with increasing odor strength [75]. This robust integration of the CSDn into the AL is supported by physiology [53], GRASP (GFP Reconstitution Across Synaptic Partners) [38], and EM microscopy in moths [76]. Perhaps even more surprising is that the CSDn expresses pre- and postsynaptic markers in both the AL and the LH [53] (Figure 3C and Figure 3D). Thus, the arbors of the CSDn may integrate locally in all of the regions to which it projects and the extent of electrotonic coupling between arbors and proximity to the spike initiation site (SIZ) may be what determines the ability of one region to affect another through modulation. CSDn processes in the AL are large diameter and may propagate well into other processes of the neuron. Its dendrites in the LH on the other hand are thin and fine and my not propagate well back into the AL or reach the SIZ.

Figure 3.

Integration of inputs in the serotonergic system. (A). A schematic showing the different brain regions that provide inputs to the raphe nuclei in mice. Most of the raphe inputs derive from non-olfactory areas, such as the hypothalamus (see text for more details). Higher-order olfactory areas such as the piriform cortex do provide input to the raphe. There are no direct inputs from the OB to the raphe and olfactory information only reaches raphe neurons indirectly through brain regions downstream of the OB. (B). In insects, the CSDns are locally integrated into each olfactory region and make reciprocal connections in each of its target structures. This allows the AL to partially regulate the release of 5-HT into the region. (C). A representation of the Drosophila brain showing a CSDn’s (green) innervation of the three primary olfactory regions in flies, the AL, the MB, and the LH. The MB receives little CSDn innervate, but both the AL and LH receive serotonergic innervation exclusively from the CSDn. (D). Pre- and postsynaptic markers are tagged with GFP and mCherry respectively and expressed in the CSDn. The CSDn displays pre- and postsynaptic markers in both the AL and the LH suggesting it integrates information locally in each olfactory region that it innervates.

As so many similarities exist between serotonergic modulation in the OB and AL, it is intriguing to ask whether OB-projecting raphe cells could also be inhibited during olfaction. Indeed, some raphe neurons are inhibited by odors [70], but it remains to be determined if such units actually project to the OB. Regardless, the mechanism for suppression or activation of OB-projecting raphe neurons would likely be different in rodents as raphe axons make only asymmetric synapses in the OB [45,77]. Inhibition of these neurons would instead need to be indirect via higher cortical regions, for example via piriform cortex interactions with the raphe. Any presynaptic regulation of raphe output in the OB would need to arise from ectopic volume transmission to have escaped detection with traditional EM.

The Role of Paracrine Signaling of 5-HT in Olfaction

While raphe axons and the CSDns make classical synapses in the OB and AL respectively, it is known that neuromodulators can also signal great distances using bulk or volume transmission [78-81]. Serotonin is found in most tissue including, the blood and CSF [82] of mammals and in the haemolymph of insects [83,84]. The DA1 pheromone-sensitive glomerulus in flies demonstrates the potential importance for such extrasynaptic 5-HT. First, PNs innervating the DA1 glomerulus (DA1-PNs) are highly sensitive to serotonergic pharmacology [5,53], yet the glomerulus receives little to no direct serotonergic innervation [38,53,73]. Killing the CSDns via expression of diphtheria toxin does not eliminate sensitivity to serotonergic pharmacology [53]. Finally, while stimulating the CSDn does not modulate DA1-PN odor responses, stimulating all serotonergic neurons in the fly together (except for the CSDns) does produce a long lasting suppression of odor responses [53] (Figure 2H). These data suggest that the relevant signal for modulation at this glomerulus may be paracrine-released 5-HT that diffuses through the haemolymph of the fly. Stimulation of the serotonergic system in the fly can indeed result in sustained 5-HT levels [85]. Similar work in Aplysia shows that neuronal activity can also increase serotonin in the haemolymph [86]. Thus, modulation at this glomerulus may be more sensitive to small increases in the activity of the total serotonergic network that allow 5-HT levels to accumulate rather than the strong activation of a small subset of serotonergic neurons innervating the AL. It will be interesting to see how other glomeruli that are more densely innervated by serotonergic fibers and not selective for pheromones respond to CSDn versus total serotonergic network stimulation. In mice, the pheromone-processing accessory olfactory bulb and the main olfactory bulb are differentially modulated by serotonin [87]. However, the role of 5-HT paracrine signaling has not been well explored in either structure.

Conclusions and Outlook

By comparing serotonergic modulation of olfactory processing across phyla, many central principles begin to emerge. First it is clear that serotonin does not target a specific class of neurons in these circuits, but rather its actions are broadly distributed across many cell types through several families of 5-HT receptors. Second, in both rodents and insects, one of the main roles of 5-HT is to indirectly suppress input at the first stage of olfactory processing. Finally, 5-HT has a non-uniform effect on output neurons in the OB and the AL, which may either increase odor discrimination or favor the perception of some odors over others.

While tremendous advancements have been made in understanding the cellular effects of 5-HT in the OB and the AL, there still remain broad unanswered questions. First is the need to correlate serotonin release in these structures with olfactory processing. Understanding what conditions result in the release of serotonin is critical for distinguishing between different models for its role in modulation. Calcium imaging studies of raphe axons in the OB should be a good approximation for serotonin release across glomeruli during olfactory tasks. Second, it will be interesting to see if serotonin serves a similar function in higher order olfactory areas or in other sensory modalities. Are the roles of 5-HT in the AL and OB unique and optimized for early olfactory processing, or will they generalize to different circuits? Finally, it is essential to consider that serotonin may have multiple functions in olfactory circuits. Most studies of modulation isolate a single feature of olfactory processing and attempt to perturb it through serotonergic modulation. This often leads to a model proposing a single function for 5-HT, which is often served by a single or few receptor classes. However, a wide variety of 5-HT receptor classes are present in the AL and OB. It is likely that this diversity in receptors imparts a diversity of function in the interaction between these two systems. At a time when techniques are creating new experimental opportunities it is exciting to realize that the advances made in olfaction will also likely be informative regarding centrifugal modulation across sensory systems and potentially other modulatory circuits.

Glossary

OR

olfactory receptor

OSN

olfactory sensory neuron

OB

olfactory bulb

AL

antennal lobe

DRN

dorsal raphe nucleus

MRN

median raphe nucleus

CSDn

contralaterally projecting serotonin-immunoreactive deutocerebral neuron

PN

projection neuron

LN

local neuron

SAC

short axon cell

ETC

external tufted cell

PG

periglomerular cell