Role of 5-hydroxytryptamine type 3 receptors in the regulation of anxiety reactions

Abstract

5-Hydroxytryptamine (5-HT) type 3 receptor (5-HT₃R) is the only type of ligand-gated ion channel in the 5-HT receptor family. Through the high permeability of Na ⁺, K ⁺, and Ca ²⁺ and activation of subsequent voltage-gated calcium channels (VGCCs), 5-HT₃R induces a rapid increase of neuronal excitability or the release of neurotransmitters from axon terminals in the central nervous system (CNS). 5-HT₃Rs are widely expressed in the medial prefrontal cortex (mPFC), amygdala (AMYG), hippocampus (HIP), periaqueductal gray (PAG), and other brain regions closely associated with anxiety reactions. They have a bidirectional regulatory effect on anxiety reactions by acting on different types of cells in different brain regions. 5-HT₃Rs mediate the activation of the cholecystokinin (CCK) system in the AMYG, and the γ‍-aminobutyric acid (GABA) “disinhibition” mechanism in the prelimbic area of the mPFC promotes anxiety by the activation of GABAergic intermediate inhibitory neurons (IINs). In contrast, a 5-HT₃R-induced GABA “disinhibition” mechanism in the infralimbic area of the mPFC and the ventral HIP produces anxiolytic effects. 5-HT₂R-mediated regulation of anxiety reactions are also activated by 5-HT₃R-activated 5-HT release in the HIP and PAG. This provides a theoretical basis for the treatment of anxiety disorders or the production of anxiolytic drugs by targeting 5-HT₃Rs. However, given the circuit specific modulation of 5-HT₃Rs on emotion, systemic use of 5-HT₃R agonism or antagonism alone seems unlikely to remedy anxiety, which deeply hinders the current clinical application of 5-HT₃R drugs. Therefore, the exploitation of circuit targeting methods or a combined drug strategy might be a useful developmental approach in the future.

1. Introduction

Anxiety is a defensive emotional response produced by the body when it suffers from various types of stressors including injuries, threatening stimuli, or acute changes in the surrounding environment. However, if the level and duration of stressors are excessive, anxiety can develop into a mental disease, namely anxiety disorder ( Crocq, 2015). Anxiety disorder manifests several types of clinical symptoms, including panic disorder (PD), generalized anxiety disorder (GAD), social anxiety disorder (SAD), posttraumatic stress disorder (PTSD), specific phobia, and obsessive-compulsive disorder (OCD), which are difficult to diagnose and treat ( Giacobbe and Flint, 2018; Vu and Conant-Norville, 2021; Lakhtakia and Torous, 2022). Meanwhile, severe psychic irritation, dysfunction of the endocrine system and immune system along with several side-effects like chest tightness, insomnia, diarrhea, and vomiting occur in anxiety disorder. Because of a considerable increase of morbidity and a younger age of onset, anxiety disorder is becoming one of the major mental diseases in clinical medicine and public health ( Giacobbe and Flint, 2018). Many animal studies and clinical practices have shown that dysfunction of the endogenous 5-hydroxytryptamine (5-HT) system is closely related to the occurrence and maintenance of anxiety and anxiety disorder. In mice, selective activation of somas of 5-HT neurons in the dorsal raphe nucleus (DRN) and median raphe nucleus (MRN) using optogenetic methods has induced obvious anxiety reactions, including novelty suppressed feeding (NSF), marble-burying behavior (MBB), and elevated-plus maze (EPM), in several emotional behavior tests ( Challis et al., 2014; Marcinkiewcz et al., 2016; Abela et al., 2020). In contrast, inhibition of 5-HT neural activation in MRN and DRN by pharmacological or genetic manipulation, or direct lesion of these areas by physical methods was found to produce anxiolytic effects ( Andrade et al., 2004). Moreover, in patch-clamp studies, the bursting number of action potential of 5-HT neurons in the MRN was significantly increased under an anxiety state ( Abela et al., 2020). Meanwhile, abundant clinical data related to selective serotonin reuptake inhibitors (SSRIs) in the treatment of anxiety disorder have been reported recently ( Hutchison et al., 2021; Murphy et al., 2021). All this evidence indicates that the mechanisms by which central nerve 5-HT systems modulate anxiety reactions are worthy of detailed study.

In central nervous system (CNS), 5-HT neurons are distributed mainly in the raphe nuclei, and send dense projections to the whole brain. The 5-HT neurotransmitters released from axon terminals of these 5-HT projections subsequently bind to 5-HT receptors (5-HTRs) to execute the modulation of brain function ( Sharp and Barnes, 2020; Ślifirski et al., 2021). At least seven subtypes of 5-HTRs have been identified (5-HT₁R‍–‍5-HT₇R), but unlike other 5-HTRs that belong to G-protein-coupled receptors (GPCRs), 5-HT₃R is the only cation channel among all these subtypes. The high permeability of 5-HT₃R to Na ⁺ and Ca ²⁺ allows it to trigger an instantaneous depolarization current and therefore mediate more rapid intracellular signaling than other 5-HTR subtypes ( Cortes-Altamirano et al., 2018; Zhao et al., 2018). In laboratory studies, 5-HT₃Rs have been suggested to have a regulatory effect on anxiety. Studies focused on 5-HT₃R-targeted specific brain regions and circuits found that 5-HT₃Rs in different regions and circuits appeared to elicit various, even conflicting, effects on behavioral expression, implying that the role of 5-HT₃Rs on the regulation of anxiety tended to be bidirectional. Meanwhile, although rigorous and precise clinical research has not yet been reported, selective intervention of the function of 5-HT₃Rs by antagonist administration has been shown to play a positive role in the treatment of anxiety disorder ( Harrell and Allan, 2003; Bhatt et al., 2013, 2017a, 2017b). By comparing clinical and laboratory studies, the lack of clinical studies seemed to be associated with the complexity of the role of 5-HT₃Rs on the regulation of anxiety in the brain. Therefore, more detailed knowledge of the effects of 5-HT₃Rs on the regulation of anxiety and the mechanisms involved is necessary to assess the potential of 5-HT₃Rs as a new therapeutic target for anxiety. In this paper, we describe the bidirectional role of 5-HT₃Rs in the regulation of anxiety and give a preliminary interpretation by depicting 5-HT₃R neural networks involved in the regulation of anxiety. This information may provide valuable guidance for more in-depth research on the role of 5-HT₃Rs in the regulation of anxiety in animals, and for exploring the potential medicinal value of 5-HT₃Rs in the clinical treatment of anxiety.

2. 5-HT₃R structure and function

5-HT₃R is the only type of ligand-gated ion channel (LGIC) among all the 5-HTR families. It also belongs to the Cys-loop receptor superfamily because of its Cys-loop structure in the N-terminal domain ( Basak et al., 2018). Analysis of its protein conformation showed that 5-HT₃R is a pseudo-symmetric pentamer formed from five transmembrane protein subunits with a central hole. When 5-HT₃Rs are activated, positive ions can pass through the central hole ( Gibbs and Chakrapani, 2021). Five classes of 5-HT₃R protein subunits, 5-HT₃RA‍–‍5-HT₃RE, have been isolated, each containing a long extracellular N-terminal domain, four transmembrane domains (TMDs), and a short extracellular C-terminal domain. There is also an intracellular domain between TMD3 and TMD4, regulating both receptor function and channel conductance. Except for 5-HT₃RD, the subunits contain six amino-acid loops close to the N-terminal domain. These amino-acid loops include the “major” A‍–‍C subunit loops and the “complementary” D‍–‍F subunit loops that constitute the binding site of 5-HT₃R ( Celli et al., 2017). 5-HT₃RA is the key protein subunit of 5-HT₃R for membrane integration. All the other types of protein subunits assemble with 5-HT₃RA to form the functional 5-HT₃R ( Barnes et al., 2009; Gibbs and Chakrapani, 2021).

The neurotransmitter 5-HT is a natural agonist of 5-HT3R, which is released mainly by 5-HT neurons derived from the MRN and DRN ( Lummis, 2012). 5-HT₃R is also regulated by other neurotransmitters or drugs. The main agonists of 5-HT₃Rs include 5-HT analogues and positive allosteric modulators (PAMs). 2-Methylserotonin and its metal salts represent 5-HT analogues, which can selectively and completely activate 5-HT₃R to exert its biological effects ( Giordano and Gerstmann, 2004). The structurally similar compounds, 1-‍(3-chlorophenyl)biguanide (mCPBG) and trans-3-‍(4-methoxyphenyl)‍- N-‍(pentan-3-yl)acrylamide (TMPPAA) are the typical PAMs frequently used in biological and preclinical studies. Both are partial agonists of 5-HT₃A protein subunits, but have a great ability to activate heterozygous 5-HT₃Rs containing both 5-HT₃A and 5-HT₃B subunits when binding to their interface, and therefore actuate allosteric regulation of 5-HT₃R responses ( Kilpatrick et al., 1990; Campbell et al., 1995; Maksay et al., 2005; Gasiorek et al., 2016). The endogenous neurotransmitter dopamine (DA) has also been shown to partially agonize 5-HT₃Rs. In electrophysiological studies, the peak responses of 5-HT₃R induced by 1 mmol/L DA were nearly one-quarter of the maximum responses induced by 5-HT ( Solt et al., 2007).

The main antagonists of 5-HT₃Rs include competitive antagonists and negative allosteric modulators. The competitive antagonists block the function of 5-HT₃Rs mainly through emulously preempting the binding sites of ligands on 5-HT₃Rs. These types of antagonists usually have three essential elements: an aromatic ring, a carbonyl-binding group, and a nitrogen atom. The frequently used competitive antagonists, tropisetron, granisetron, bemesetron (MDL7222), and ondansetron, all have the above characteristics. These antagonists have not only been widely used in antagonistic studies of 5-HT₃Rs, but also clinically used in the treatment of nausea, vomiting, and other diseases related to 5-HT₃R dysfunction in cancer patients ( Bhatt et al., 2021). Other synthetic 5-HT₃R competitive antagonists have also been synthesized in recent laboratory studies. These include (4-benzylpiperazin-1-yl)‍(3-methoxyquinoxalin-2-yl)methanone (6g) and N- n-propyl-3-ethoxyquinoxaline-2-carboxamide (6n), which are both synthesized by condensation and chlorination of o-phenylenediamine and ketol diethyl malonate ( Bhatt et al., 2013, 2017a, 2017b). Kurhe et al. (2015, 2017) synthesized 3-methoxy- N- p-tolylquinoxalin-2-carboxamide (QCM-4) based on the structure of 6n. The main negative allosteric modulators of 5-HT₃R in the natural world include terpene menthol, citronellol, and geraniol. Chemically separated substances like capsaicin, cannabidiol, and gingerol also show negative allosteric modulation of 5-HT₃R. They interfere with 5-HT₃R functions by conformational change (al Kury et al., 2018; el Nebrisi et al., 2020).

Electrophysiological studies have shown that 5-HT₃Rs have high permeability for cations like Na ⁺ and K ⁺ and can quickly regulate the excitability of neurons ( Yakel et al., 1990; Machu, 2011; Gibbs and Chakrapani, 2021). Activated 5-HT₃Rs transform the Na ⁺/K ⁺ permeability ratio of neurons from 0.05 under resting state to 0.92‍–‍0.94 under active state, inducing rapid inward cation currents, and therefore directly raising the membrane potential of neurons. These rapid inward cation currents induced by 5-HT₃Rs can further activate voltage-gated calcium channels (VGCCs) leading to a rapid influx of extracellular Ca ²⁺ and the elevation of neurotransmitter release from axonal terminals of neurons. These effects have been proven to mediate the quick release of γ‍-aminobutyric acid (GABA) in inhibitory interneurons (IINs) in hippocampus (HIP) slices ( Yakel and Jackson, 1988; Turner et al., 2004). In addition, laser scanning confocal imaging studies showed that 5-HT₃Rs themselves have a specific permeability to Ca ²⁺. Administration of mCPBG onto mouse hippocampal slices increased the intracellular Ca ²⁺ level directly through 5-HT₃R functions, and thereby enhanced the release of neurotransmitters at the presynaptic level ( Choi et al., 2007; Fawley et al., 2019). In conclusion, the modulation of neuronal activity by 5-HT₃Rs includes two main aspects. First, 5-HT₃Rs rapidly up-regulate neuronal excitability through Na ⁺ influx. Second, 5-HT₃Rs directly or indirectly (through activation of VGCCs) mediate an increase of the intracellular Ca ²⁺ level to enhance the release of neurotransmitters in axon terminals, including 5-HT, GABA, and cholinergic and adrenergic neurotransmitters ( Zhao et al., 2018). Further research of cellular sublocalization of 5-HT₃Rs showed that 5-HT₃Rs expressed on the postsynaptic membrane or cell body mediated mainly the rapid up-regulation of neuronal excitability, while 5-HT₃Rs expressed on the presynaptic membrane mainly regulated the release of presynaptic neuronal transmitters ( Sharp and Barnes, 2020).

5-HT₃Rs are widely expressed in the central, peripheral, and enteric nervous systems, playing an important role in the regulation of various nervous activities, including gastrointestinal peristalsis, digestive juice secretion, vomiting, neuroimmunity, visceral reflex, pain response, and emotional response ( Cortes-Altamirano et al., 2018; Juza et al., 2020; Irving et al., 2021). In early studies, 5-HT₃Rs were detected in both the forebrain and brain stem of humans‍—tissues closely related to anxiety reactions ( Kilpatrick et al., 1987; Barnes et al., 1990). Ondansetron injection into the rat brain can significantly increase activation of neurons in these regions, indicating a potential relationship between 5-HT₃Rs and anxiety ( Urzedo-Rodrigues et al., 2014). By using 5-HT₃AR-green fluorescent protein (GFP) transgenic mice combined with immunofluorescence technologies, Koyama et al. (2017) found that circuit-based distribution of 5-HT₃R positive signals in several brain areas closely associated with anxiety, such as the prefrontal cortex, HIP, and amygdala (AMYG), providing more direct and precise morphological evidence to support the study of the modulatory effects of 5-HT₃R on anxiety behavior.

3. Bidirectional effects of 5-HT₃Rs on anxiety behaviors

There have been several studies of the association between the regulation of anxiety and emotional behavior. Although treatment strategies targeting 5-HT₃Rs are still in the elementary stages, some clinical practices and pharmacological and genetic experiments have shown that 5-HT₃R antagonists have appreciable or at least auxiliary anxiolytic effects. In clinical studies, several types of 5-HT₃R competitive antagonists like tropisetron, zatosetron, ondansetron, and granisetron, have been proven to alleviate the unhealthy emotions of patients of different gender and ages suffering from anxiety disorders ( Smith et al., 1999; Haus et al., 2000; Harmer et al., 2006). Evaluation of the therapeutic effect on OCD has shown that tropisetron and ondansetron can reverse the compulsive behavior of OCD patients to some extent, and this effect can be enhanced by combinational administration of fluvoxamine ( Hewlett et al., 2003; Soltani et al., 2010). In another clinical study, after 12 weeks of oral ondansetron treatment, 64.3% of OCD patients showed reversal of anxiety ( Pallanti et al., 2009). Granisetron has also been shown to have a rapid therapeutic effect in moderate or severe OCD patients ( Askari et al., 2012; Serata et al., 2015; Sharafkhah et al., 2019). 5-HT₃R antagonists have also been used in symptomatic improvement of GAD and SAD patients ( Lecrubier et al., 1993; McCann et al., 1997). After administration of tropisetron for GAD or ondansetron for SAD, anxiety symptoms were obviously alleviated. Currently, there is no evidence from clinical studies of other types of anxiety disorder to support a regulatory role of 5-HT₃Rs, but animal models suggest that 5-HT₃Rs may be a potential target in the management and treatment of anxiety disorder.

Many animal studies have shown that specific antagonism of 5-HT₃Rs produces anxiolytic-like reactions. EPM is one of the classic models used to test anxiety level in the laboratory. Because its open arms can directly generate anxiety stimuli in the behavioral testing of mice, the anxiety level of animals can be detected without extra stressors or stimuli in the EPM test. Cutler (1991) and Artaiz et al. (1995) introduced granisetron orally into male DAB rats and by intraperitoneal injection into male Wistar rats. In the EPM test, significant increases in the number of animals entering open arms and the time spent in open arms were observed, indicating anxiolytic effects. Intraperitoneal injection of MDL7222 into ACI/N rats can also increase the staying time and entering frequency in open arms in the EPM test ( Hensler et al., 2004). The anxiolytic-like behaviors induced by antagonistic effects on 5-HT₃R have also been extensively reported in different anxiety-model animals. In studies related to the effects of 6g and 6n on emotional regulation, Bhatt et al. (2013, 2017a, 2017b) tested different types of anxiety disorder models, including lipopolysaccharide (LPS), chronic unpredictable mild stress (CUMS), and traumatic brain injury (TBI), using male Swiss Albino mice. Both 6g and 6n produced a significant reduction of anxiety-like responses on all three models measured by EPM, open-field test (OFT), and light-dark box test (LDT). Kurhe et al. (2017) also detected a significant reduction in anxiety level using QCM-4, a derivative of 6n, in a high-fat diet (HFD)‍-induced anxiety mouse model. Consistent with the above antagonistic experimental results, mice treated with mCPBG by intracerebroventricular injection to activate cerebral 5-HT₃Rs produced a significant anxiety-like response in EPM. A significant reduction in the staying time and entering frequency in open arms was induced ( Gupta et al., 2016), confirming the anxiolytic effects induced by antagonizing 5-HT₃Rs from the opposite side. In recent years, genetic evidence for the regulation of anxiety responses by 5-HT₃Rs has also been found. On the one hand, when human serotonin type 3 receptor ( Htr3), the primary gene coding 5-HT₃Rs, is knocked out, mice showed a significant reduction in the base level of anxiety responses in several testing models including EPM and LDT ( Bhatnagar et al., 2004). On the other hand, overexpression of 5-HT₃Rs in the murine brain through targeted follicle-stimulating hormone ( FSH) gene knockout caused the experimental mice to exhibit anxiety-like behaviors ( Bi et al., 2020). In conclusion, the above studies have confirmed that 5-HT₃Rs have promoting effects on anxiety responses from both pharmacological and genetic aspects, i.‍e., activating 5-HT₃Rs induces anxiety-like behaviors, whereas antagonizing 5-HT₃Rs weakens the anxiety response or significantly lowers the base level of response to anxiety stimuli in normal individuals.

However, some studies did not support the boosting effects of 5-HT₃R on the regulation of anxiety, as they found an anxiolytic-like effect of 5-HT₃Rs on the modulation of anxiety. For example, by using transgenic mice with overexpressed 5-HT₃Rs, Harrell and Allan (2003) observed significant reductions in the base level of anxiety in mice tested by EPM. Nowicki et al. (2014) also found that after the administration of ondansetron in the water environment of zebrafishes feeding for a period of time, their base level of anxiety response was significantly improved, indicating that 5-HT₃Rs produced anxiolytic effects. These contradictory findings were even more pronounced when the effects of 5-HT₃Rs were restricted to a specific brain region. Considering that 5-HT₃Rs have extensive distribution in multiple brain regions of the CNS and the complexity of the role of the CNS in the regulation of anxiety ( Koyama et al., 2017), the above contradictory findings likely arose because the regulation of the anxiety response by 5-HT₃Rs involves many neural circuits and cellular mechanisms. Therefore, it is necessary to classify and discuss the neuropharmacological mechanisms of 5-HT₃R action in different brain regions and neural circuits in relation to anxiety reactions.

4. Mechanism of anxiety regulation in brain circuits by 5-HT₃R

In the CNS, anxiety reactions involve complex neural mechanisms. Firstly, at the anatomical level, anxiety reactions involve the complicated simultaneous activation of multiple brain regions, including the medial prefrontal cortex (mPFC), ventral HIP (vHIP), basolateral amygdala (BLA), central amygdaloid nucleus (CeA), bed nucleus of the stria terminalis (BNST), hypothalamus, periaqueductal gray (PAG), parabrachial nucleus (PBN), and other nearby regions that have been confirmed to respond to anxiety stimuli in a previous review ( Tovote et al., 2015). Enormous neural networks are formed from connective projections between adjacent brain areas involved in the regulation of anxiety reactions. Meanwhile, the interconnections of adjacent brain areas always contain excitatory and inhibitory projections involved in anxiety reactions, giving some brain regions, like the AMYG and mPFC, both anxiogenic and anxiolytic behavioral effects ( Tovote et al., 2015). Moreover, functionally inhibitory (largely GABAergic) interneurons in local brain areas also have efficient regulatory effects on both excitatory and inhibitory outputs, and thereby influence anxiety reactions with opposite results ( Babaev et al., 2018). Therefore, to discuss the mechanism of 5-HT₃Rs in regulating anxiety, the brain regions and neural circuits controlling anxiety reactions regulated by 5-HT₃Rs should first be confirmed. Then, the regulatory role of 5-HT₃Rs in each specific brain region should be discussed separately. Moreover, the inconsistent regulation of 5-HT₃Rs in both excitatory and inhibitory projections, and interneurons executing contrary effects of anxiety reactions in some specific brain areas should also be considered.

4.1. Regulatory brain circuits of anxiety reactions related to 5-HT₃Rs

According to morphological studies by Koyama et al. (2017), the AMYG and neural circuits connecting with this brain region including the vHIP, mPFC, and PAG are considered vital targets of 5-HT₃R in response to anxiety reactions. The functional activities of these brain areas in anxiety reactions include two main aspects. First, the AMYG receives environmental stimuli from sensory inputs or primary-process emotional signals from the mPFC, integrates and encodes aversive information to downstream effector regions in the brain stem such as the PAG, one of the brainstem structures vital in controlling the “fight to flight” reaction, and thereby generates avoidance and freezing. On the other hand, it sends excitatory projections to the vHIP and subsequently activates pyramidal neurons projecting to the mPFC, and thus organizes AMYG-vHIP-mPFC neural circuits ( Fig. 1) to precisely process and embellish anxiety information at the forebrain level. Serotonin projections, large resources of 5-HTs in the CNS derived from the DRN and MRN, have been clearly shown to generate connections to these brain regions. Therefore, 5-HT₃Rs can be directly activated by stimulation of 5-HTs released from terminals of these projections during anxiety reactions to give play to the regulatory roles of anxiety processing. In the following section, we discuss the mechanism of the regulation of anxiety by 5-HT₃Rs in the AMYG, vHIP, mPFC, and PAG.

Fig. 1. Schematic diagram of main brain regions participating in the regulation of anxiety by 5-hydroxytryptamine (5-HT) type 3 receptors (5-HT₃Rs). The black arrows represent neuronal connections between brain regions. The yellow arrows represent 5-HT projections from the midbrain. mPFC, medial prefrontal cortex; vHIP, ventral hippocampus; PAG, periaqueductal gray; AMYG, amygdala; DRN, dorsal raphe nucleus; MRN, median raphe nucleus.

4.2. Mechanism of 5-HT₃R regulation of anxiety in the AMYG

The AMYG was one of the first brain regions found to be closely associated with anxiety reactions. Human clinical studies and analysis of immediate early genes in rodents have demonstrated a significant increase in AMYG activity under anxiety conditions. Pharmacological studies have also shown that damaging the bioactivity of AMYG neurons leads to a significant anxiolytic response in EPM of animals, which further confirms the important role of the AMYG in the regulation of the anxiety reaction ( Sah, 2017). In clinical neuroimaging studies, the AMYG was considered to be closely associated with many types of anxiety disorder. In some magnetic resonance imaging (MRI) studies, abnormally increased volumes of the AMYG were observed in OCD and GAD patients ( Szeszko et al., 1999; Millan, 2022). Unusually and selectively activated patterns of the AMYG were also observed in SAD patients when exposed to fear or aversive conditions ( Birbaumer et al., 1998). In PTSD, the AMYG was considered the center of neural circuits involved in the regulation of fear conditioning, while hyperactivation of the AMYG was confirmed by functional brain meta-analyses in adult PTSD ( Rogan et al., 1997). In short, the AMYG plays a vital role in the regulation of anxiety. Its abnormal hyperactivation is a commonly observed clinical situation in many types of anxiety disorder.

The AMYG is mainly composed of the BLA and CeA, which establish the main neural pathway for the occurrence and maintenance of the anxiety response in the AMYG. The BLA is the vital region of acceptance of anxiogenic information in the AMYG. It can receive excitatory sensory inputs related to environmental changes and cue-related behaviors from cortical association areas like the mPFC, and then processes and integrates anxiety information on glutamatergic projections in this area with high efficiency. This information can be transmitted into GABAergic inhibitory projections of the CeA to inhibit the activity of downstream brain regions like the PAG and cause fear or disgust ( Babaev et al., 2018). In the regulation of loop circuits in the AMYG, the projective glutamatergic neurons of the BLA send projections into IINs in the CeA and a population of intercalated cells (ITCs) that are also GABAergic interneurons located in an intercalated island, a region sandwiched between the BLA and CeA. IINs in the CeA project mainly into GABAergic projection neurons in this region that negatively regulate the excitatory projections derived from the BLA to the CeA. In contrast, ITCs between the BLA and CeA project mainly to IINs located in the CeA to action their feedforward inhibitory regulation. There are also some IINs located in the BLA. Their terminals can administer the projective glutamatergic neurons of the BLA-innervating ITCs and the cell bodies of ITCs in the intercalated island, to inhibit the activity of ITCs. Thus, a complex information transmission network ( Fig. 2) of anxiety reactions is formed in the AMYG ( Babaev et al., 2018).

Fig. 2. Schematic paradigm of the regulation of anxiety by 5-hydroxytryptamine (5-HT) type 3 receptors (5-HT₃Rs) in the amygdala (AMYG). The BLA receives excitatory inputs encoding anxiogenic information from cortical association areas like the mPFC, and then innervates four main excitatory projections: (1) innervates to the CeA inhibitory projecting neurons to induce inhibition of the CeA on downstream brain regions like the PAG and produces anxiogenic effects; (2) innervates to CeA IINs to inhibit BLA excitatory projections innervating CeA inhibitory projecting neurons and produces anxiolytic effects; (3) innervates to ITCs to inhibit CeA IINs and induces "disinhibitory" effects on BLA excitatory projections innervating CeA inhibitory projecting neurons to evoke anxiogenic effects; (4) innervates the vHIP to become involved in the organization of AMYG-vHIP-mPFC neural circuits. 5-HT₃Rs are expressed mainly on the CCK positive IINs in the BLA innervated by 5-HT projections derived from the MRN and DRN, which have high efficiency to induce CCK release, thereby activating CCK-2Rs expressed on BLA excitatory projection fibers innervating ITCs and ITCs themselves, i.e., activating ITCs to promote anxiogenic effects. CeA, central amygdaloid nucleus; BLA, basolateral amygdala; mPFC, medial prefrontal cortex; PAG, periaqueductal gray; vHIP, ventral hippocampus; MRN, median raphe nucleus; DRN, dorsal raphe nucleus; CCK, cholecystokinin; CCK-2R, CCK-2 receptor; ITCs, intercalated cells; IINs, inhibitory interneurons; EP, excitatory projection; IP, inhibitory projection; SP, serotonin (5-HT) projection.

In the feedback regulatory network of anxiety reactions in the AMYG, 5-HT₃Rs play their anxiety regulatory role mainly through the activation of the cholecystokinin (CCK) system. CCKs are important neuropeptides in the AMYG involved in promoting and maintaining anxiety responses. Injection of CCK analogs into the AMYG can induce anxiety-like behaviors or enhance anxiety responses in experimental animals, while antagonism of CCK function can significantly reduce anxiety responses (del Boca et al., 2012; Holm et al., 2014). CCK-positive neurons in the AMYG are mainly IINs co-expressed with GABA in the BLA. Immunofluorescence and in situ hybridization studies showed that 5-HT₃Rs are densely expressed on the cell body and axon terminals of CCK-positive neurons. Meanwhile, these areas also receive 5-HT projections derived from the MRN and DRN ( Mascagni and McDonald, 2007). Consequently, these CCK-positive neurons can directly receive excitatory information from the raphe nucleus during anxiety reactions to activate 5-HT₃Rs and thereby induce Ca ²⁺-dependent release of CCKs ( Uvnäs-Moberg et al., 1996). In the CNS, CCKs bind mainly to their specific CCK-2 receptors to play their biological role. CCK-2 receptors are widely distributed in the neurons innervated by IINs in the BLA, including BLA glutamatergic projection fibers innervating ITCs and ITCs themselves in the intercalated island. Thus, CCKs can further induce K ⁺-dependent release of GABA at the postsynaptic level through a coordinated mechanism with glutamate receptors by activating CCK-2 receptors on these cells. This results in hyperpolarized potentials from BLA glutamatergic projections innervating ITCs as well as ITCs themselves and decreased excitability of these neurons observed in patch-clamp studies (de la Mora et al., 2007). These CCK system-mediated effects reduce the inhibitory regulation of ITCs on IINs in the CeA, and thus produce a “disinhibitory” effect on the excitatory projections of the BLA to the CeA, i.e., they indirectly enhance the neural transmission of this pathway. Therefore, in the neuronal micro-loop regulation of the AMYG, 5-HT₃Rs inhibit the inhibitory regulation of ITCs on IINs in the CeA by promoting the release of CCKs and subsequently enhancing CCK-2 receptor-mediated GABA release, thus indirectly enhancing the excitatory projection of the BLA to CeA and promoting the occurrence and development of anxiety reactions ( Fig. 2). In a pharmacological behavioral study in vivo, administration of tropisetron or ondansetron into the AMYG of male BKW mice resulted in a significant decrease in the level of anxiety reactions in LDT ( Costall et al., 1989). Higgins et al. (1991) also found significant anxiolytic-like responses after injection of ondansetron, tropisetron, and MDL7222 into the AMYG of Lister Hooded rats, proving the efficacy of 5-HT₃Rs in the promotion of anxiety reactions in the AMYG ( Fig. 2). Considering the unidirectional contributory role of 5-HT₃Rs to anxiety in the AMYG, together with abnormal hyperactivation of the AMYG represented in anxiety disorders, interception of the inhibition of 5-HT₃Rs in anxiolytic pathways seems to be a relatively effective means to weaken pervasive classes of anxiety disorder, such as PTSD, SAD, GAD, and OCD. Thus, in future the AMYG could become a hypothetically ideal target in anxiety treatment oriented toward clinical 5-HT₃R antagonists.

However, it is still unclear if 5-HT₃Rs are expressed on IINs or ITCs to directly regulate excitatory anxiogenic projections from the BLA to the CeA, so the morphological and mechanistic aspects of 5-HT₃Rs in regulation of anxiety in IINs and ITCs of the AMYG remain to be further explored.

4.3. Mechanism of 5-HT₃R regulation of anxiety in the mPFC

The neurons in the mPFC are involved mainly in processing attention and emotion-related information. Therefore, the mPFC is a critical brain area in the CNS to regulate the fear extinction response and aversive conditioning. In pathology, dysfunction of the mPFC is associated with PTSD. Compared to healthy subjects, PTSD patients show a decreased gray matter volume in the mPFC ( Koch et al., 2017). OCD is also linked to the abnormal structure and function of the mPFC. In morphology, a voxel-based morphometry study revealed a significant reduction of gray matter volume in the bilateral mPFC of OCD patients compared with that of healthy subjects ( Togao et al., 2010). The mPFC in rodents is composed mainly of the infralimbic cortex (IL) area and prelimbic cortex (PL) area, both of which have close fiber connections with the AMYG. However, they have completely opposite effects on the functional regulation of anxiety reactions. The PL mainly receives excitatory inputs from the vHIP and processes and transmits processed information to activate the downstream BLA, which can further enhance the BLA-CeA-mediated fear/disgust emotional response pathway and induce and promote anxiety reactions. By contrast, the IL contains a large number of ITCs that receive excitatory input from the PL and then send inhibitory projections to the BLA. This plays a regulatory role in the extinction of fear response by inducing a certain degree of anxiolytic effect ( Motzkin et al., 2015; Park and Moghaddam, 2017). The utterly different regulatory effects of PL and IL on anxiety reactions lead to the entirely opposite results of 5-HT₃R in the regulation of the emotional response in the mPFC ( Fig. 3).

Fig. 3. Schematic paradigm of the regulation of anxiety by 5-hydroxytryptamine (5-HT) type 3 receptors (5-HT₃Rs) in the mFPC. The PL receives excitatory inputs processing anxiogenic information from the vHIP, and then innervates excitatory projecting neurons to the BLA, producing anxiogenic effects. Meanwhile, it also innervates excitatory collaterals to IL inhibitory projections innervating to the CeA inhibitory projecting neurons, to inhibit CeA inhibitory projecting neurons to produce anxiolytic effects. 5-HT₃Rs are densely distributed in both the PL and IL IINs innervated by 5-HT projections derived from the MRN and DRN. Activating 5-HT₃Rs expressed on PL IINs can promote GABA release to inhibit PL excitatory neurons projecting to the BLA, and thereby produces anxiolytic effects. In contrast, activating 5-HT₃Rs expressed on IL IINs will inhibit IL inhibitory neurons projecting to the CeA through the same GABA mechanism, to induce anxiogenic effects. PL, prelimbic cortex; vHIP, ventral hippocampus; BLA, basolateral amygdala; IL, infralimbic cortex; CeA, central amygdaloid nucleus; IINs, intermediate inhibitory neurons; MRN, median raphe nucleus; DRN, dorsal raphe nucleus; GABA, γ‍ -aminobutyric acid; EP, excitatory projection; IP, inhibitory projection; SP, serotonin (5-HT) projection.

In situ hybridization studies showed that the positive signals of 5-HT₃R mRNA were extensively co-expressed with GABAergic interneurons in the PL and IL. There was also a high 5-HT input innervated by the DRN and MRN in these two regions. These findings show that 5-HT systems have the ability to directly modulate GABAergic inter-inhibitory neurons in both the PL and IL via 5-HT₃R activation ( Gui et al., 2010). By enhancing GABA release through activation of 5-HT₃R, the inhibitory inputs of mPFC pyramidal neurons are enhanced, and thereby hyperpolarization is induced and excitability is reduced ( Gui et al., 2010). These molecular effects were further confirmed in the study of Riga et al. (2016), in which administration of vortioxetine and ondansetron into the mPFC significantly increased the excitability of pyramidal neurons and the release of glutamate. These electrophysiological effects were blocked by SR57227A. From the above, corresponding to the opposite effects of the PL and IL on the regulation of anxiety reactions, activation of 5-HT₃R distributed in the PL can produce anxiolytic effects, whereas activation of 5-HT₃R distributed in the IL seems to promote anxiety ( Fig. 3). Given the above evidence from animal experi‍ments suggesting that 5-HT₃Rs expressed on the mPFC elicit bidirectional effects on anxiety, a precise clinical management is required to target mPFC 5-HT₃Rs, which seems difficult to achieve under current clinical systems. Thus, more basic information is still needed, especially of the difference in expression of 5-HT₃Rs between the PL and IL, and of the level of activation in these two areas under anxiety conditions, to guide the development of precise methods for treating anxiety disorders in clinical practice.

4.4. Mechanisms of 5-HT₃R regulation of anxiety in the vHIP

The HIP is one of brain regions in the CNS that plays an important role in regulating both cognitive learning and emotional responses. It has been well demonstrated that dysfunctions of HIP are obviously linked to anxiety disorders, especially PTSD. Several clinical reports and neuroimaging meta-analyses have indicated that the hippocampal volume of chronic PTSD patients was smaller than that of healthy controls ( Bremner et al., 1995, 2021). The HIP can be anatomically divided into the dorsal (dHIP) and ventral (vHIP) parts ( Tseilikman et al., 2022). In relevant reviews, Bannerman et al. (2004) and Engin and Treit (2007) have pointed out that the dHIP may be more involved in the processing and disposal of a series of information inputs related to learning and memory, while the vHIP may be specifically responsible for the regulation of anxiety-related functions. Therefore, the vHIP is a focus in current research on the mechanism of anxiety reactions. In the neural connections among brain areas during anxiety reactions, the vHIP can receive excitatory inputs from the BLA ( Fig. 2), and processes fear or disgust information encoded by the BLA. Then the vHIP sends excitatory projections to the PL in the mPFC, which has downstream excitatory inputs to the BLA ( Fig. 4). Thus, as an important relay for the transmission and processing of disgust/fear information, the vHIP has strong positive regulatory effects on anxiety reactions ( Tovote et al., 2015).

Fig. 4. Schematic paradigm of the regulation of anxiety by 5-hydroxytryptamine (5-HT) type 3 receptors (5-HT₃Rs) in the vHIP. The vHIP pyramidal neurons receive excitatory inputs encoding anxiogenic information derived from the BLA, and then send excitatory projections to the PL in the mPFC, thereby becoming involved in the establishment of the AMYG-vHIP-mPFC circuit to promote anxiety reactions. 5-HT₃Rs are expressed on vHIP IINs and presynaptic terminals of 5-HT projections. Activation of 5-HT₃Rs expressed on vHIP IINs can promote GABA release to inhibit pyramidal neurons to produce anxiolytic effects. In contrast, activation of 5-HT₃Rs expressed on presynaptic terminals of 5-HT projections can secondarily promote 5-HT₂Rs expressed on pyramidal neurons, thereby facilitating the AMYG-vHIP-mPFC circuit to promote anxiety. vHIP, ventral hippocampus; BLA, basolateral amygdala; PL, prelimbic cortex; mPFC, medial prefrontal cortex; IINs, intermediate inhibitory neurons; GABA, γ‍ -aminobutyric acid; AMYG, amygdala; Pyr, pyramidal neuron; MRN, median raphe nucleus; DRN, dorsal raphe nucleus; EP, excitatory projection; IP, inhibitory projection; SP, serotonin (5-HT) projection.

Like the mPFC, 5-HT₃Rs are also expressed mainly on IINs in the vHIP. Thus, activation of 5-HT₃Rs in the vHIP leads to Ca ²⁺-dependent GABA release from these interneurons, which increases inhibitory inputs and decreases excitability of pyramidal neurons ( Fig. 4). Consequently, the excitatory inputs to the PL from the vHIP will be reduced, resulting in anxiolytic effects ( Riga et al., 2016). A genetic study by Harrell and Allan (2003) confirmed this view. After overexpression of the gene encoding 5-HT₃R in the HIP, mice showed greater exploratory behavior in response to stimuli in novel environments and a significant anxiolytic response in the EPM. Likewise, while the expression level of 5-HT₃Rs was significantly reduced in the HIP through targeted knockout of neuroplastin 65 protein, mice exhibited a significant reduction in the time that they entered the central area in the OFT test, i.e., the knockout produced anxiety reactions ( Li et al., 2019). This study confirmed the anxiolytic effects induced by 5-HT₃Rs in the vHIP from the opposite side.

However, some pharmacological behavioral studies do not support the induction of anxiolytic effects by hippocampal 5-HT₃Rs. Stefański et al. (1993) did not observe a significant occurrence of anxiety response in OFT tests after injection of ondansetron and tropisetron into the HIP of rats. Meanwhile, Vogel’s test showed that rats produced anti-anxiety/depression-like effects after the same operations. Although no studies have directly investigated the cause of this contradiction, 5-HT₃R-induced 5-HT release could be a possible explanation. In addition to IINs in vHIP local areas, 5-HT₃Rs are also expressed on the terminals of 5-HT projections derived from the DRN and MRN to the vHIP. Activation of 5-HT₃Rs expressed on these fiber terminals can promote the release of Ca ²⁺-dependent 5-HTs ( Martin et al., 1992). Although these effects could produce a certain degree of positive feedback on 5-HT₃Rs, a large amount of 5-HT₂Rs, other subtypes of 5-HTRs mediating the excitatory G protein pathway, are expressed in the cell body of pyramidal neurons in the vHIP and could also be positively activated by the release of Ca ²⁺-dependent 5-HTs promoted by 5-HT₃Rs ( Mendelson and McEwen, 1991). If this positive feedback loop is continuously activated, the release of 5-HTs will escalate and excitatory effects on pyramidal neurons mediated by 5-HT₂Rs may become dominant. Thus, the rapid increase in the excitability of pyramidal neurons mediated by 5-HT₂Rs will positively promote the vHIP-PL-BLA anxiety regulatory pathway and promote the occurrence and development of an anxiety response ( Alves et al., 2004). The anxiolytic effects of 5-HT₃Rs on GABA release in the vHIP may be counterbalanced or even reversed by activation of 5-HT₂Rs, and thereby produce anxiogenic effects ( Fig. 4). In summary, the vHIP might be another target of bidirectional effects of 5-HT₃Rs involved on anxiety. However, unlike the mPFC, the 5-HT₃Rs-involved bidirectional effects on anxiety in the vHIP are presumably attributed to the interaction between 5-HT₃Rs and 5-HT₂Rs, whereas 5-HT₃Rs themselves tend to unidirectionally contribute to anxiolysis in the local circuit. Because of the relatively close relationship between the HIP and PTSD, an effective therapeutic strategy targeting 5-HT₃Rs in the vHIP against PTSD might be a combination of the 5-HT₃R agonist with the antagonist 5-HT₂R.

4.5. Mechanism of 5-HT₃R regulation of anxiety in the PAG

PAG is a gray matter structure surrounding the midbrain aqueduct. In addition to being involved in the transmission and regulation of pain information, PAG has positive regulatory effects on anxiety reactions. In PAG, IINs receive inhibitory projections from the CeA and have a tonic inhibitory effect on excitator y neurons (mostly vesicular glutamate transporter 2 (vGluT2)‍-positive neurons) in this region ( Tovote et al., 2016). Since vGluT2-positive neurons have the ability to promote anxiety reactions ( Taylor et al., 2019), the inhibitory projections derived from the CeA to PAG can inhibit the IINs in PAG to “disinhibit” vGluT2-positive neurons, thereby inducing anxiety reactions ( Tovote et al., 2016). Whether 5-HT₃Rs are expressed on IINs in PAG is unknown. Behavioral tests showed that administration of mCPBG into PAG did not cause significant changes in the anxiety level of mice during EPM ( Lopes et al., 2022). Thus, the 5-HT₃Rs may not directly regulate the anxiety reactions in PAG. However, 5-HT₃R antagonists could significantly inhibit the anxiolytic effect mediated by the 5-HT₂R agonist 1-(3-chlorophenyl)piperazine (mCPP) ( Lopes et al., 2022). The 5-HT₂Rs are expressed mainly on the soma and dendrites of IINs throughout the PAG, induce GABA release from IINs, and antagonize the action of vGluT2-positive neurons to produce an anxiolytic reaction ( Griffiths and Lovick, 2002; Tovote et al., 2016; Vilela-Costa et al., 2021). Meanwhile, 5-HT₃Rs have a similar distribution of 5-HT projection terminals (from the DRN and MRN) in PAG as in the vHIP ( Lopes et al., 2022). Therefore, 5-HT₃Rs may be involved in anxiolytic responses by enhancing Ca ²⁺-dependent 5-HT release in PAG, thereby activating 5-HT₂Rs and promoting GABA release to induce GABA-controlled anxiolytic responses ( Fig. 5). In pathology, PAG is closely associated with PD. Electrical stimulation of the brain using stereotactic neurosurgery in the PAG of humans can induce intense states of fear and/or terror ( Zwanzger et al., 2012). Further research is needed to provide a theoretical foundation for exploring the prospect of a potential application to PD treatment targeting 5-HT₃Rs.

Fig. 5. Schematic paradigm of the regulation of anxiety by 5-hydroxytryptamine (5-HT) type 3 receptors (5-HT₃Rs) in the PAG. PAG IINs receive inhibitory inputs from the CeA to relieve tonic inhibition of their downstream excitatory projections, thereby becoming positively involved in anxiety reactions through a "disinhibition" mechanism. 5-HT₃Rs are expressed mainly on presynaptic terminals of 5-HT projections here, and efficiently promote activation of 5-HT₂Rs located in PAG IINs when activated. Activation of 5-HT₂Rs promotes the inhibition of downstream excitatory projections to produce anxiolytic effects. PAG, periaqueductal gray; IINs, intermediate inhibitory neurons; CeA, central amygdaloid nucleus; MRN, median raphe nucleus; DRN, dorsal raphe nucleus; EP, excitatory projection; IP, inhibitory projection; SP, serotonin (5-HT) projection.

5. Conclusions

5-HT₃Rs have bidirectional regulation effects on anxiety reactions due to the various characteristics of their distribution in different brain regions and cell types in the CNS. In the AMYG, 5-HT₃R-mediated release of CCK and subsequent “disinhibition” of the BLA-CeA pathway are essential mechanisms for 5-HT₃R-induced promotion of anxiety reactions. In contrast, 5-HT₃R-mediated GABA release and inhibition of the vHIP-PL/PL-BLA pathway are the main mechanism responsible for anxiogenic effects in the vHIP and PL, and are also efficient in promoting anxiety in the IL. Besides, the positive regulation of 5-HT release by 5-HT₃R in the vHIP and PAG, and subsequent enhancement of 5-HT₂R functional activity suggest that 5-HT₃Rs may regulate anxiety reactions through the action of 5-HT₂Rs, facilitating anxiety in the vHIP and generating anxiogenic effects in PAG. There is evidence that certain anxiety subtypes impact mainly distinct local brain circuitry, which suggests that systemic intervention with 5-HT₃Rs would have therapeutic potential. For instance, 5-HT₃R antagonists would be more efficient for GAD and SAD for targeting mainly in the AMYG, whereas 5-HT₃Rs agonists could be auxiliaries in PD therapy in view of the anxiolytic effects of 5-HT₃Rs on PAG. However, the circuit specific and bidirectional regulation of 5-HT₃Rs in controlling emotions makes it difficult to directly apply 5-HT₃R agonists or antagonists to generally treat or prevent anxiety. Therefore, exploiting circuit targeting methods and applying a combined medication strategy might be effectual in increasing the therapeutic values of 5-HT₃Rs on anxiety disorders involving multiple regions of the brain. Furthermore, the expression of 5-HT₃Rs on a specific subtype of neuron in different brain regions is not yet clear, and whether 5-HT₃Rs have interactions with other 5-HTR subtypes involved in anxiety is also unknown, and remains to be addressed in future.

Author contributions

Yinan DU: conceptualization, writing ‒‍ original draft, writing ‒‍ review & editing, and project administration. Zhiwei LI: investigation and visualization. Yukui ZHAO: investigation and visualization. Jing HAN: writing ‒‍ review & editing and funding acquisition. Weiping HU: supervision and funding acquisition. Zhiqiang LIU: supervision, writing ‒‍ review & editing, project administration, and funding acquisition. All authors have read and approved the final version.

Compliance with ethics guidelines

Yinan DU, Zhiwei LI, Yukui ZHAO, Jing HAN, Weiping HU, and Zhiqiang LIU declare that they have no conflict of interest.

This review does not contain any studies with human or animal subjects performed by any of the authors.