Ominous Odors: olfactory control of instinctive fear and aggression in mice

Abstract

Aggression and fear are often thought to be distinct behavioral states, yet they share several common output responses. In the mouse, both can be initiated by specialized odor cues. How these cues signal through the olfactory system to promote behavior is largely unknown. Recent experiments have started to uncover the relevant signaling ligands, chemosensory receptors, and responsive sensory neurons that together enable the precise manipulation of behaviorally relevant neural circuits. Moreover, the use of molecular genetics and new experimental strategies has begun to reveal how the central nervous system processes olfactory information to initiate aggression and fear. A sensory-initiated comparative study of these two fundamental threat reactions promises to offer new mechanistic insight.

Introduction

Animals constantly encounter threats. Whether the menace is a competitor of the same species or a hungry predator, initiating an appropriate aggressive or fearful response increases the individual’s chance of survival and reproduction. These critical behaviors are displayed similarly across species, which suggests that they are generated by evolutionarily-conserved neural mechanisms, as Darwin first recognized well over a century ago [1]. Decades later, Cannon summarized the intricate relationship between the two main threat response states, fear and aggression, as “fight or flight” [2]. Even though both fear and aggression have been extensively studied [3], their underlying neural mechanisms remain largely unknown.

Fighting or fleeing may be promoted by stimuli that have been previously associated with threat or by specialized cues that have intrinsic meaning. Associative fear has been intensively studied [4]. This paradigm has two distinct facets: the generation and recall of a memory as well as the display of fear. Innate fear presents an alternate and simplified model to focus studies on behavioral output. Olfactory cues command instinctive behavior with especially high probability in a wide variety of animals [5–8]. The mouse detects specialized odorants emitted by conspecifics and heterospecifics, termed pheromones and kairomones, respectively, which are sufficient to robustly trigger the release of innate fearful and aggressive behaviors [8,9]. Thus, olfaction can be experimentally leveraged to control, manipulate, and investigate the downstream neural circuits and mechanisms that promote innate behavior.

Fear and aggression are each complex behavioral states that have proven difficult to “solve” independently. However, when considered together, their similarities and differences may provide mechanistic understanding (Figure 1). At the circuit level, both require an intact amygdala and hypothalamus [10,11]. Through neuroendocrine modulation, both tip the autonomic balance to sympathetic domination. At the level of motor output the two behavioral states may even seamlessly integrate, as when a cornered animal suddenly turns “fear into fury” [2]. However, the study of fear and aggression is often approached separately because of numerous differences in output: aggression is offensive (but see [12] for a more detailed discussion), fear is defensive; aggression promotes piloerection and rearing to increase apparent size, fear results in freezing and huddling to dissuade attention. Furthermore, fear and aggression are under differential developmental control. For example, territorial aggression is not elicited before the onset of puberty, whereas predator odor induces fear in juveniles [13,14]. These similarities and differences provide a comparative framework to identify the common neural correlates that enable mobilization to threats, as well as the particular mechanisms that distinguish their separate but related motor patterns.

Figure 1. Pheromones and kairomones influence fear and aggression in mice.

Defensive responses to olfactory stimuli can be aggression-specific (blue), fear-specific (red), or shared by both fearful and aggressive behaviors (purple). The shared response primarily includes activation of the hypothalamic-pituitary-adrenal axis (HPA) axis, which, for example, increases heart rate and decreases digestive secretions to deal with a threat. Facial expressions [61] and other motor patterns [27] differ in fearful and aggressive contexts.

Specialized odor cues promote fear and aggression in the mouse

Almost any odorant can be conditioned to elicit fear or aggression. In addition, the mouse detects a subset of specialized odors that elicit behavior without prior associative conditioning. While the sources of many of these odorants are known, such as male mouse urine or predator excretions, the identities of most of the bioactive ligands remain unidentified. The ability of these cues to evoke aggressive and fearful responses across individuals without previous experience and learning indicates that the underlying neural correlates are in part genetically determined, which may provide experimental traction. For example, these specialized ligands activate unknown subsets of sensory receptors in the olfactory system, but once such receptors are identified, they can be used to study the neurons that are functionally relevant to aggression and fear.

Mice detect odors through two primary olfactory sensory organs: the main olfactory epithelium (MOE) and the vomeronasal organ (VNO) (Figure 2a). The MOE mediates both associative olfactory learning and instinctive responses, while the VNO is considered specialized for innate behaviors. However, sensory neurons housed in both the MOE and VNO promote innate aggression and fear [8,9,15–20]. Mutant mice lacking sensory detection in either the MOE or VNO have dramatically reduced aggression [15,16,21]. Similarly, mice lacking sensory detection in either the dorsal region of the MOE or the VNO have severe reductions in freezing and avoidance to various predator odor sources such as fox feces and cat fur, and even an anesthetized rat [8,19,20,22]. Olfactory sensory neurons generally express one out of approximately 1400 G-protein coupled odorant receptors, which tunes their response to a subset of odors. These odorant receptors include roughly 1000 olfactory receptors (ORs) and 15 trace amine associated receptors (TAARs) in the MOE, as well as over 350 vomeronasal receptors (VRs) in the VNO [23–25]. Complex odor sources such as those emitted by other animals will likely activate many distinct olfactory sensory neurons. With so many possible sensory signals, how is the mouse olfactory system organized to translate a particular sensory response into a reliable reaction such as attack or fleeing? To investigate this, several groups have begun to isolate the odorant sources and in some cases the specific ligands that initiate these stereotyped behaviors as well as their cognate receptors and sensory neurons.

Figure 2. Overlapping and distinct areas of the brain and modulating factors involved in pheromonal control of fear and aggression in mice.

(a) Main areas of the brain and periphery that have been implicated in olfactory-mediated unconditioned fear (red), aggression (blue), or both (purple), as determined by lesions, microstimulation, and immediate early gene (IEG) expression. Areas shown in grey are only implicated by anatomy. Although braces depict the forebrain regions receiving direct projections from both second order olfactory areas, most anatomical details between brain regions are not shown for clarity. Many areas are bidirectionally connected through both excitatory and inhibitory projections. This gross anatomical view neglects heterogeneous cell types and subcircuits within areas, and is based on experiments in both rats and mice. Additionally, non-olfactory contextual information must enter the circuits through various routes, some of which are shown with dashed grey arrows. (b) Many factors dynamically regulate information flow through the circuits in (a) and thus affect the probability of olfactory-mediated fear and aggression over several timescales.

Significant headway in elucidating the identity of individual receptors tuned to aggression- and fear-inducing odors has now been made in both the MOE and VNO. Isogai et al. performed a comprehensive investigation into the functional specificity of nearly 90 VRs [22]. They exposed mice to ethologically relevant ligand sources such as conspecific and heterospecific bedding material and correlated sensory neural activity with the functional identity of the VR. They found many VRs activated selectively by either female or male mouse odors (including putative aggression-promoting cues), others activated by odors from other species (including those of putative predators such as rats, snakes, and birds), and a minority activated by both conspecific and heterospecific stimuli. This last class of sensory neurons could form the basis of responses that are shared in both fear and aggression (e.g. increased sympathetic outflow [26]), while the male-conspecific and heterospecific specialized sensory neurons may stimulate effectors that underlie the differences between the behaviors [27].

In another study, Ferrero et al. took an alternative approach by screening for biologically relevant ligand sources that activate TAARs, non-canonical odorant receptors in the MOE [28]. They found heterologously expressed TAAR4 to be activated by urine from mountain lions and bobcats. Further, 2-phenylethylamine, a volatile compound enriched in carnivore urine, activates TAAR4 and promotes avoidance behavior as well as increases in corticosterone stress hormone circulation. Thus, two distinct classes of odorant receptors operating in separate olfactory structures, TAARs and VRs, detect cues that induce aggression and fear. It is not known how distinct signals from the MOE and VNO ultimately integrate with information from other sensory modalities including vision, audition, and touch, to promote behaviors such as attack or freezing. The identification of the relevant odorant receptors provides a molecular handle to genetically label, visualize, and ultimately manipulate the downstream neural circuits.

Circuits mediating aggressive and fearful behaviors: overlapping and distinct

Aggressive and fearful behaviors are complex motor sequences that typically employ the coordination of multiple muscle groups, and yet they are highly stereotyped across mice [11,29], implying the existence of “hard-wiring” to tie the coordinated actions together. While recent work has revealed many of the details of associative fear circuits [4,30–33], the identification of the neurons in the central nervous system that are necessary and sufficient for instinctive fear and aggression, how they are genetically specified, and how they fit into the overall control of behavior in an animal remains a challenge. Recently, several excellent reviews have independently considered the circuits underlying a single threat behavior [27,10,11,3], but the relationships between fear, aggression, and the olfactory system have not been jointly explored.

Following sensory stimulation of either the MOE or VNO, which excite the main olfactory bulb (MOB) and accessory olfactory bulb (AOB), respectively, the circuits mediating fear or aggression become less apparent as the anatomy of second order projections becomes more complicated [34,35] (Figure 2a). The circuit involves other brain regions that have been functionally identified through lesions, microstimulation, and IEG expression. These gross anatomical regions display both overlapping and mutually exclusive functional contributions to fear and aggression. At this stage even the anatomical details of the threat-responsive circuits have not been well established and the neural mechanisms are largely unknown. However, as the signals progress closer to the motor periphery a more concrete understanding of the regions driving these behaviors emerges again. Functional studies show that the periaqueductal gray (PAG) plays a key role in the generation of fear-related motor patterns [11]. Perhaps the best-studied effector output involved in both fear and aggression is the secretion of corticotropin-releasing hormone and vasopressin by parvocellular neurons of the hypothalamic paraventricular nucleus (PVN) in order to kick off the hypothalamic-pituitary-adrenal axis (HPA) axis response to stress [36]. This output can be engaged simply by an olfactory stimulus such as conspecific urine [26]. However, since afferent projection patterns to the PVN are highly complex, little is known about how olfactory information releases the HPA response.

Several groups have recently compiled a detailed anatomy of MOB and AOB projections in the mouse [37,38,34]. Further work has highlighted the preponderance of bidirectional connections between many areas [39,40] and the overlap of projections from the main and accessory olfactory systems [35]. One particular study [41] employed engineered viruses to reveal that mitral cells localized in the dorsal MOB preferentially project their axons to the cortical amygdala, raising the tantalizing possibility that this topography directs MOE-initiated instinctive responses [19,20]. Such anatomical work is essential to determine the focus of mechanistic studies that will bridge the gap between behavioral and molecular understanding. Accordingly, Choi et al. [42] revealed important anatomical details of the projection from the medial amygdala (MEA), which is widely innervated by the AOB, to the ventromedial nucleus of the hypothalamus (VMH), an area implicated in both innate fear and aggression. They found molecularly-distinct populations of excitatory and inhibitory neurons within the MEA that differentially project to regions of the VMH implicated in separate innate behaviors. These results provide testable models of how these circuits may interact to “gate” output. Bian and colleagues [43] expanded these results by combining anatomical tracing with slice electrophysiology to highlight possible cellular mechanisms of olfactory-mediated fear and aggression. They show that the VMH-projecting neurons in the MEA have unexpected “cortical-like” properties such as ion channels that allow burst coding, thus inviting future studies of the behavioral significance of such mechanisms.

Recently, Lin et al. utilized IEG analysis, optogenetics, and electrophysiology to confirm the ventrolateral part of the VMH (VMHvl) as a critical node in the aggression-producing circuit [44]. By taking a genetic approach, this investigation disambiguated classical studies of microstimulation-induced aggression in other animals, which could have recruited fibers of passage, and extended their striking results to the mouse model. The dorsomedial part of this same nucleus (VMHdm) has also been implicated in unconditioned olfactory fear responses by IEG expression analyses [42,45]. Interestingly, photostimulation of the VMHdm resulted in freezing and fleeing [44], suggesting a spatial segregation of aggression and fear motor commands in the VMH. Does the proximity of aggression- and fear-related neurons in the VMHvl and VMHdm, respectively, contribute to the shared threat response elicited by pheromones and kairomones or the seamless switch from “fear to fury”? The potential intermingling of relevant neurons highlights the need for sophisticated approaches to verify and investigate the function of specific microcircuit connections.

Genetic and molecular control of fear and aggression

Many of the brain regions implicated in fear and aggression are functionally and molecularly heterogeneous. The absence of methods to disentangle them has impeded a mechanistic study of the microcircuits involved. However, knowledge of the developmental mechanisms and neuromodulatory influences underlying innate aggression and fear may be exploited to dissect and delineate the relevant neural circuits with genetic techniques. Indeed, this strategy has been elegantly demonstrated in studies of the neural pathways governing reproduction in nematodes, courtship in fruit flies, and the regulation of feeding behavior in mice [46–48].

In addition to being elicited without experience, a striking feature of many instinctive behaviors is their tendency to vary dramatically and stereotypically over several timescales according to an animal’s age, sex, state, or various other factors (Figure 2b). For instance, olfactory-initiated aggressive behavior is elicited with much higher probability in adult versus juvenile or castrated males, as well as in lactating versus virgin females [9,14,21]. In mammals, sex hormones released from the gonads during both development and adulthood play a fundamental role in directing the display of aggression. More specifically, a male-specific conversion of testosterone to estrogen at birth masculinizes neural circuits that promote male-typical aggressive behaviors, while subsequent testosterone signaling “amplifies” these circuits in the adult [49–51]. Steroid responsive estrogen and androgen receptors, as well as aromatase, are expressed in sexually dimorphic, partially overlapping patterns in subcortical nuclei implicated in aggressive behavior [49,50]. Thus, these genes may delineate a functionally unified neural pathway, and offer a logical entry point for a genetically guided deconstruction of aggression circuits.

To identify additional molecular components underlying sexually dimorphic behavior, Xu et al. examined gene expression differences in males compared to females with microarray and in-situ hybridization techniques [52]. They identified a cohort of 16 genes differentially expressed according to gender in the hypothalamus and MEA, regions previously implicated in the display of aggressive behavior. Loss-of-function mutants were obtained for four of the identified genes, and behavioral testing implicated two sex hormone responsive genes, Bombesin receptor subtype-3 (Brs-3) and Insulin receptor substrate-4 (Irs-4), in the control of aggression. It is currently unclear how Brs-3 or Irs-4 act to control attack behavior, or to what extent the additional genes regulate aggressive behavior in general. Future work aimed at clarifying the cellular actions and potential epistatic interactions of the identified genes, as well as the functional roles of the neurons that express them, will likely advance our understanding of how the nervous system generates sex-specific behaviors such as odor-mediated aggression.

Can a similar approach yield insight into fear behavior? Although predator odor fear is not considered a canonical male or female typical behavior, hormonal and sex effects have been reported [13,53]. Traction in dissecting fear circuits may also come from considering additional developmental programs, hormones, and neuromodulatory influences. For instance, Choi et al. performed amygdalar gene expression analyses to find molecular markers that demarcate neural populations responsive to predator odor [42]. Studies in rats have also begun to detail how corticosterone may control the emergence of conspecific predator odor fear in juveniles as well as how noradrenergic release in the dorsal premammillary nucleus may modulate the aversive response to cat odor [54,55]. Considering these modulatory systems will likely provide an entry point towards investigating olfactory initiated fear circuits.

Concluding Remarks

Even in a simplified experimental environment, a mouse faces an uncertain and complicated world, and thus any behavior may be viewed as flexible, probabilistic, and dynamic. Although the abovementioned genetic approaches offer a clear entry point to deconstruct the relevant neural circuits, it is worth noting that a large body of work has suggested that additional molecular and neuromodulatory sources [10,56–59] and even the birth of new olfactory cells [60] contribute to the control of aggression and fear. Understanding how all of these elements ultimately converge to instruct a mouse whether to fight or flee will be necessary to achieve a systems level description of behavior (Figure 2). Amidst such complexity, the olfactory system offers an excellent opportunity to use clearly defined sensory stimuli and genetic tools to identify, manipulate and study the mechanisms that promote threat-initiated survival behavior.

Highlights.

• Olfactory cues promote innate fear and aggression

• Candidate aggression and fear promoting odorant receptors have been identified

• Overlapping and distinct brain regions mediate innate threat behaviors

• Genetics underlying innate threat responses may offer mechanistic insight

Abbreviations

AHA

anterior hypothalamic area

AOB

accessory olfactory bulb

BAOT

bed nucleus of the accessory olfactory tract

BNST

bed nucleus of the stria terminalis

COA

cortical amygdala

LA

lateral amygdala

MEA

medial amygdalar nucleus

MOB

main olfactory bulb

MOE

main olfactory epithelium

MPOA

medial preoptic area

NLOT

nucleus of the lateral olfactory tract

PAA

piriform-amygdalar area

PAGdl

periaqueductal grey, dorsolateral column

PFC

prefrontal cortex

PMD

dorsal premammillary nucleus

PMV

ventral premammillary nucleus

PVN

hypothalamic paraventricular nucleus

SC

superior colliculus

VMHdm

ventromedial hypothalamus, dorsomedial portion

VMHvl

ventromedial hypothalamus, ventrolateral portion

VNO

vomeronasal organ