Evolution of the Genetic and Neural Architecture for Vertebrate Odor Perception

Abstract

Evolution sculpts the olfactory nervous system in response to the unique sensory challenges facing each species. In vertebrates, dramatic and diverse adaptations to the chemical environment are possible because of the hierarchical structure of the olfactory receptor (OR) gene superfamily: rapid growth or pruning across the OR gene tree accompany major changes in habitat and lifestyle; independent selection on OR subfamilies can permit local adaptation or conserved chemical communication; and genetic variation in single OR genes among thousands can alter odor percepts and behaviors driven by precise chemical cues. However, this genetic flexibility contrasts with the relatively fixed neural architecture of the vertebrate olfactory system, whose slower rate of divergence mandates that new olfactory receptors integrate into segregated and functionally-distinct neural pathways. This organization allows evolution to couple critical chemical signals with selectively advantageous responses, but also constrains relationships between olfactory receptors and behavior. The coevolution of the OR repertoire and the structure of the olfactory system therefore reveals general principles of how the brain solves specific sensory problems and how it adapts to new ones.

Olfaction, the sense of smell, has evolved to detect signals from the chemical environment, which contains clues about where to move, what to eat, when to reproduce, and which stimuli to remember as rewarding or dangerous [1, 2]. How an animal responds to chemical cues is in part learned and in part innate, depending on how the olfactory nervous system has been shaped by both experience across a lifetime and evolution across generations.

Interacting with the chemical world presents a challenge unlike other sensory tasks. In contrast to light and sound waves, which vary continuously in wavelength and amplitude, chemical compounds differ discretely in an enormous number of dimensions. Compared to vision and audition, olfaction requires many more receptors – proteins expressed in sensory organs that convert a physical event into an electrochemical signal carried by neurons. A single photoreceptor pigment may interact with a range of wavelengths that is sufficiently informative for an animal, as with our dark-vision rhodopsin; however, biophysical constraints restrict chemoreceptor proteins to interact with only subsets of chemical space. The high dimensionality of chemical space and the diversity of the chemical stimuli an animal might encounter have therefore selected for genomes that encode enormous receptor repertoires, containing hundreds to thousands of olfactory receptor (OR) genes [3–6].

Here we examine how the olfactory receptor repertoire and the structure of the olfactory nervous system evolve in concert to sense and interpret chemical information. Adaptations in the genetic and neural architecture of olfaction reflect the unique chemosensory challenges faced along a species’ lineage — detecting novel odorants, discerning especially critical odors with high acuity, and responding behaviorally to molecules that acquire new meaning. We argue that the receptors underlying vertebrate olfaction possess two properties essential for the range of adaptations seen in vertebrate olfactory systems: a flexible and hierarchical pattern of evolution that allows receptor adaptation to both dramatic and subtle changes in the chemical environment; and access, through specific expression patterns, to a diverse array of neural pathways that govern both hardwired, instinctual behaviors as well as more flexible odor learning. In this way, OR families and subfamilies have evolved on larger scales to inform the animal about wide swaths of chemical space, while at the same time some individual receptors have become highly tuned to key odorants that elicit innate responses. We speculate that the unique ability to incorporate new and evolving OR genes has produced the substantial genetic and structural diversity of olfactory systems seen across the vertebrate lineage [7–9].

The Molecular, Cellular and Neural Architecture of Vertebrate Olfaction

Smell begins when odor molecules bind to OR proteins on the endings of sensory neurons. The set of odors an animal can detect therefore depends on the expression pattern and the protein structure encoded by each of its OR genes. In vertebrates, nearly all OR genes encode seven-pass transmembrane G-protein coupled receptors (GPCRs) that, upon ligand binding, signal through G-proteins and intracellular second messengers to ultimately open membrane ion channels; this depolarizes the sensory neuron to drive action potentials that are conducted along its axon into the olfactory bulb of the brain (Figure 1) [10].

Figure 1.

Organization of the mouse olfactory system (partially adapted from Dulac & Torello, 2003). Odors are blends of molecular compounds inhaled into the nasal cavity, where they interact with olfactory receptor proteins expressed in the main olfactory epithelium (medium gray), the vomeronasal organ (dark gray), or one of several smaller sensory structures not pictured (upper). Each sensory neuron expresses a single olfactory receptor denoted by its color, and neurons expressing the same receptor project to insular structures in the olfactory bulb called glomeruli (lower). Glomeruli corresponding to a given receptor have stereotyped spatial positions across animals. Mitral cells in the olfactory bulb each send a dendrite into one glomerulus (main olfactory bulb) or multiple glomeruli corresponding to the same olfactory receptor (accessory olfactory bulb, not pictured), and project axons to a variety of central brain regions that mediate odor learning and innate odor-driven behaviors. The targets of main and accessory bulb mitral cell projections are largely distinct.

In vertebrates, most olfactory sensory neurons express a single olfactory receptor gene from one of five major GPCR families [11]. The evolutionary history of each family relates roughly to the region of chemical space it probes, as the odorant structures bound by family members are reminiscent of those bound by the specific ancestral non-olfactory GPCR from which each family derived (detailed below.) In the rodent olfactory system where they were first discovered, members of each OR family are expressed in a Pointilistic pattern within spatially restricted zones of the nasal epithelia. Most ciliated sensory neurons of the main olfactory epithelium (MOE) express receptor genes of the largest family, the “classical” mammalian olfactory receptors (mORs) [12, 13], while a minority of neurons instead express members of the much smaller trace amino-acid receptor (TAAR) family [14]. In the vomeronasal organ (VNO), microvillar sensory neurons in the apical epithelial layer express Vomeronasal type 1 Receptors (V1Rs) [15], but neurons in the basal layer each express one Vomeronasal type 2 Receptor (V2R) plus a chaperone V2R in characteristic combinations [16, 17] [18]. Finally, some neurons in both layers of the VNO instead express members of the small Formyl Peptide Receptor-related family (FPRs) [19, 20] (Figures 1, 2).

Figure 2.

Phylogenetic relationships of the vertebrate GPCR olfactory receptor repertoire. A representative sample of ~2700 GPCR functional OR genes from fish (Danio rerio), frog (Xenopus laevis), mouse (Mus musculus), and human (Homo sapiens) genomes was used to construct a phylogenetic tree (cf. Manzini & Korsching 2011). Notable features are visible, including large expansions of Class II ORs (red) in tetrapods, V2Rs (purple) in amphibians, TAARs (blue) in fishes, and V1Rs (teal) in mammals; significant losses of functional ORs in humans (blue dots) compared to mice (green dots); the monophyly of Class I (orange) and Class II (red) tetrapod ORs; and the subfamily structure of the “classical” ORs, with numeric labels corresponding to the subfamilies defined in Hayden et al. (2010). A Class II OR subfamily (2,13) itself contains a smaller subfamily, the “OR37” receptors, which has atypically expanded and been conserved at the protein sequence level in humans. Note that all mammalian and some frog Class I ORs form a monophyletic clade within the larger set of Class I ORs and other fish OR subfamilies that are neither Class I nor Class II.

The one-receptor-per-neuron organization shapes how odor representations propagate through the olfactory system. The axons of primary sensory neurons expressing the same receptor gene coalesce onto insular structures called glomeruli in the outer layer of the olfactory bulb, with the positions of the glomeruli corresponding to a given OR approximately fixed from one animal to the next [21] (Figure 1). A given odorant binds to a stereotyped subset of olfactory receptors and thereby activates a characteristic subset of glomeruli – like keys pressed together as a chord on a piano – such that odors elicit unique spatiotemporal activity patterns across the olfactory bulb [22].

Odor responses in the olfactory bulb drive innate and learned behaviors via neural projections to more central brain regions. Mitral and tufted (MT) cells of the main olfactory bulb each send a single dendrite into one glomerulus, and extend parallel axon branches into multiple higher olfactory targets, including the piriform cortex – which promotes associations between odors and rewards or punishments – and the cortical amygdala – which mediates instinctual responses to some predator odors (Figure 1) [23–25]. In contrast, mitral cells in the accessory olfactory bulb each extend a dendrite into multiple glomeruli corresponding to the same VR and project to amygdalar, hypothalamic, and basal forebrain nuclei involved in reproductive and stress responses (Figure 1) [26]. The targets of main and accessory bulb mitral cells are mostly distinct, and thus higher-order olfactory brain regions have likely adapted to receive information about odors sensed by different olfactory receptors that release distinct behaviors.

Olfactory brain regions, especially the olfactory bulb, vary tremendously in relative size across vertebrate species; furthermore, in many species, the accessory olfactory bulb and the vomeronasal organ are absent [7, 8]. We therefore focus primarily on the conserved genetic architecture of vertebrate olfaction, which likely arose in the common ancestor of all ray-fined fishes and tetrapods (lived ~460-430 MYA). The genome of the zebrafish contains multiple members of four of the five major GPCR OR families (ORs, TAARs, V2Rs, and several V1Rs) (Figure 2) [27–29] and its sensory neurons generally express these receptors in the same “one receptor, one sensory neuron” pattern as in mice [30–32]. Zebrafish sensory neurons project to insular glomeruli within the olfactory bulb, where innervation from different sensory neuron types (expressing distinct OR families) is at least partially segregated [33, 34]. Although the pattern of convergence from sensory neurons to glomeruli has not been assessed for most species, this conservation across zebrafish and rodents suggests that the neural organization of the olfactory system forms a stable background on which to compare the changes in OR gene repertoire. The molecular mechanism for choosing a single OR to be expressed in each sensory neuron – which engages both cell type-specific transcription factors as well as stochastic DNA-DNA interactions [11] – seems to have directly linked the genetic evolution of OR families with the functional evolution of odor perception. This process has allowed the olfactory system to flexibly incorporate or discard chemosensors, as each OR gene stakes out its own population of sensory neurons in the olfactory epithelium and its own glomerular territory in the olfactory bulb without disrupting the expression or function of other receptors.

The Birth-and-Death Mechanism of Gene Family Evolution Allows Specific Adaptation to Chemical Space

Singular OR expression potentially affords each newly arising OR gene a neural pathway to inform the brain about the odorants it binds. Evolution of the OR gene repertoire should therefore reflect the salient features of an animal’s chemical environment. To test this hypothesis, genomic observations of OR evolution must be combined with knowledge of what molecules individual receptors bind and how different animals respond to these odors. Unfortunately, because of the enormous number of OR proteins and the difficulty of measuring their molecular tuning in vitro, we know only a small fraction of OR-odorant interactions [35].

Nevertheless, thanks to the mechanism of OR repertoire evolution, it is possible to compare finer portions of the OR gene tree as they grow or die off on multiple evolutionary timescales. The membership of olfactory receptor families evolves through a characteristic birth-and-death process. Changes in the number of OR genes result from duplications of contiguous family members during unequal meiotic crossing over, resulting in a redundancy and functional independence of the new genes that allows them to mutate and acquire new ligand-binding properties [36]. A side-effect of the relaxed selection pressure on recently duplicated genes is the accumulation of nonsense or frameshift mutations that terminate receptor function, producing pseudogenes embedded in rapidly duplicating OR gene clusters [37]. This birth-and-death process further explains the low proportion of orthologous receptors found among species, as many new OR genes can arise in the time since the divergence of even closely related lineages [38]. Other modes of gene family evolution have occasionally homogenized the OR repertoire, such as gene conversion between tightly-linked subfamily members and across-species conservation of single ORs that function outside the olfactory system [38–40]; however it is clear that birth-and-death and relaxed selection are the dominant processes by which different species evolve markedly different sets of olfactory receptors [41, 42].

The birth-and-death mechanism allows natural selection to act independently at multiple levels of the OR gene tree, including each of the five olfactory GPCR families, more recently diverged subfamilies within each OR family, and individual OR genes (Figure 2). We argue below that progressively more subtle adaptations in the habitat, lifestyle, and behavior of vertebrate lineages correspond to increasingly fine-scale growth and death on branches of the OR tree. We then discuss how the olfactory nervous system might take advantage of these changes in the receptor repertoire to generate new innate and learned olfactory behaviors.

Olfactory Overhaul: Widespread Reshaping of the OR Repertoire in New Sensory Environments

As some vertebrates have adapted to new habitats or shifted to lifestyles that rely less on smell, entire OR families and olfactory sensory organs have become dispensable. For example, as its visual system has expanded dramatically, the old world primate lineage has seen a massive loss of functional olfactory receptor genes, as well as a near total loss of the vomeronasal receptor repertoire, accompanied by pseudogenization of an important VNO chemotransduction channel gene TrpC2 (Figure 2) [43, 44]. Selection on primate OR genes may have relaxed following the duplication of the opsin gene encoding the long-wavelength retinal photopigment, as the genomes of trichromat primates – all old world monkeys and one species of new world monkey, the howler monkey – have significantly higher ratios of pseudogenized to intact OR genes than dichromat new world monkeys [45]. Trichromacy may have allowed color vision to take over some olfactory tasks, like discriminating the ripeness of leaves by their appearance along the red-green axis [46, 47]. This view is controversial, however, because the proportion of OR pseudogenes in a mammalian genome does not correlate with the number of intact ORs and thus poorly estimates olfactory function [38]; moreover, loss of intact ORs in primates may have begun before the divergence of new world monkeys and old world monkeys [48]. Nevertheless, the growth of the visual nervous system in old world primates likely reduced reliance on olfaction for many social and foraging behaviors, which may represent a general pattern: in multiple cases the enhancement of other senses like vision, echolocation, and electroreception coincides with a smaller OR repertoire [49] [50].

Changes in habitat that broadly alter chemical ecology appear to have also resized and reshaped the OR families dramatically, though directly demonstrating causal relationships remains a challenge. The divergence of the tetrapod lineage from its aquatic relatives ~420 MYA was followed by an order-of-magnitude expansion in the major vertebrate OR gene family (represented by the mORs in mammals), as hundreds to thousands of these “classical” OR genes are found in individual amphibian and terrestrial vertebrate genomes, whereas fish genomes have, at most, 150 (Figure 2) [51]. The idea that having more OR genes is adaptive on land is strengthened by converse evidence from multiple independent mammalian lineages that returned to aquatic life. In these animals (e.g., whales, seals, and manatees), the number of intact OR genes has dropped since they diverged from their terrestrial relatives (e.g. cattle, dogs, and elephants) [52].

The classical OR gene family has two major branches, the so-called Class I and Class II ORs; however, only the Class II branch shows a massive increase in membership as a possible result of adapting to land. This receptor subfamily is expansive in the genomes of tetrapods (amphibians and terrestrial vertebrates) but is essentially absent in fishes, including one more closely related to tetrapods than teleosts – the coelacanth (Figure 2) [53, 54]. Why might a particular OR subfamily have expanded in a given vertebrate lineage? The simplest hypothesis is that the ancestral family members were useful for sensing type of chemical structure common or critical in that animal’s chemical environment. Having more OR genes with somewhat diversified ligand-binding regions could have provided a selective advantage – for instance by increasing perceptual range, sensitivity, or discriminative power in this region of chemical space. The evolutionary history and functional tests of the distinction between Class I and Class II ORs support this model. First identified in the amphibian Xenopus laevis genome, Class I genes cluster phylogenetically with fish ORs, while Class II genes cluster with the majority of mammalian ORs (Figure 2) [55]. Furthermore, Class I genes are expressed exclusively in the lateral diverticulum of the frog nasal cavity, which senses waterborne but not airborne odorants from the outside world, while Class II genes are expressed only in the medial diverticulum, which opens only when the frog is in air [55]. Functional characterization supports the hypothesis that Xenopus Class I genes are “fish-like”, as when expressed in heterologous cells they detect highly water-soluble ligands like charged amino acids – important water-borne chemosignals for fish. Conversely, Class II Xenopus ORs detect some airborne volatile compounds like the aroma of coffee [56]. The ligands of mammalian Class I ORs are also on average more polar (and thus more water-soluble) than Class II ligands [35]; but the relative size of the Class I repertoire has remained stable across mammals, suggesting that even in purely terrestrial environments these receptors bind odorants of ecological significance [38]..

Skipping the Classics: Particular Habitats May Have Favored Expansion of Distinct OR Families

The presence of smaller “classical” OR families in aquatic animals does not necessarily mean that olfaction is less important in water than in air. Similar to the above cases of Class I and Class II ORs, the aquatic environment may have instead favored the use of other OR families because their ancestral ligand-binding properties gave them a head-start in detecting more common aqueous odorants. For instance, the TAAR family has expanded rapidly in teleost fishes (Figure 2). The zebrafish genome encodes over 100 functional TAARs, almost an order of magnitude more than any vertebrate outside the teleost clade. This expansion includes the derivation of an entirely new TAAR subfamily under strong positive selection in the teleost lineage, which was first thought not to encode amine receptors (like the other TAARs) because a canonical amine-binding motif is lost in many subfamily members [27]. It has now been shown, however, that these teleost TAARs initially evolved a second amine-binding motif, changing their specificity to diamines over monoamines; later, a subset of these secondarily lost the original motif, resulting in TAARs that use a distinct mechanism for amine binding [57]. Together this pattern of TAAR duplication and diversification suggests that the outsized importance of amine chemosensation drove this particular OR family to prominence in the teleost lineage. This notion is consistent with the important role known to be played by amino acids and their diamine decarboxylation products in the chemical ecology of zebra fish [58].

The ancestral TAAR was likely the orthologue of the mammalian TAAR1 and was therefore probably a receptor for internally produced trace monoamines like TAAR1 ligands tyramine, octopamine, and tryptamine [59]. Thus, the expansion of the TAAR family in teleosts may have been biased from the start, since only a few mutations in new TAAR genes would allow better sensing of amines, a region of chemical space apparently important for the behavior of these fishes. For example, cadaverine, a diamine signature of putrefaction, is detected by the zebrafish-specific receptor TAAR13c [58] and elicits innate aversion. This link between a novel receptor and amine-driven behavior may be a lineage-specific adaptation, as another teleost, the goldfish, is attracted to cadaverine (perhaps via the action of distinct TAAR receptors) (Figure 3) [57, 58, 60].

Figure 3.

A sample of monomolecular odorants that drive innate, species-specific behaviors across vertebrates and insects. Instinctual reproductive, protective, and feeding behaviors are released by a variety of chemical structures, including volatile airborne small molecules in rodents (top row); waterborne acids, bases, and peptides in fishes and amphibians (middle row); and volatile hydrocarbons in insects (bottom row). Chemical structures (receptor responsible for behavior, if known), left to right: top row 2,4,5-trimethythiazoline, 2-sec-butyl-4,5- dihydrothiazole [SBT], 2,3-dehydro-exo-brevicomin [DHB] [76], phenethylamine (Mus musculus TAAR4) [84], trimethylamine (Mus musculus TAAR5) [85]; middle row 4-hydroxyphenylacetic acid (Danio rerio ORA1 [a V1R]) [95], cadaverine (Danio rerio TAAR13c) [58], splendiferin (peptide pheromone for Litoria splendida), sodefrin (peptide pheromone for Cynops pyrrhogaster) [63]; bottom row methylhexanoate (D. sechellia Or22a), octanoic acid [78], cis-11-vaccenyl acetate (D. melanogaster Or67d) [79], (10E,12Z)-hexadeca-10,12-dien-1-al [upper] and (10E,12Z)-hexadeca-10,12-dien-1-ol [lower] (aliases bombykal, receptor Bombyx mori BmOr3, and bombykol, receptor B. mori BmOr1, respectively) [80]. Behavioral responses are sometimes enhanced by combinations of odorants, as with SBT and DHB in mice and bombykal and bombykol in the silk moth. Moreover, responses to the same compound(s) may differ drastically between species or between sexes of the same species.

The evolutionary trajectory of the TAARs raises the possibility that that along different lineages, specific ancestral receptors have “gotten lucky” by binding chemical structures similar to those with wider ecological significance. This line of reasoning could explain why the V2R family expanded dramatically and reached its largest membership in the amphibian lineage (Figure 2) [61]. The V2Rs are derived from an ancestral GPCR that senses the amino acid glutamate and have diversified to allow broader detection of amino acids and longer polypeptides [17]. The amphibian V2R family is expressed in the amphibian vomeronasal organ, which plays an essential role in sensing waterborne peptide sex signals and coordinating complex mating behavior [62, 63].

Given these examples suggesting that the ecological importance of a given class of chemicals is reflected by the size of the OR family that detects that chemical class, it is intriguing to consider its converse: whether the apparent expansion of a specific OR family can reveal the types of chemical cues that are most relevant to a given species. Consider the case of the semiaquatic platypus, one of only two extant members of the monotreme lineage, which was initially thought not to rely on strong olfactory abilities given the high proportion of pseudogenes in its classical OR family and its use of a unique electroreceptive “bill sense” for locating prey in brackish water; however, it was later found that the platypus genome contains more functional V1R family members than any other vertebrate, reflecting an unparalleled expansion of this particular gene family in the monotreme lineage [64]. Indeed, the platypus possesses one of the most complex vomeronasal organs described, suggesting that odor detection through the “accessory” olfactory system has become its main mode of chemosensation. Because V1Rs detect many steroid hormones that regulate reproductive and mating behavior [65], combing this region of chemical space may reveal cues critical to the behavior of the platypus and ecological reasons that this particular receptor family expanded.

The Finer Things in Life: Olfactory Specialization Through OR Subfamily Selection

OR birth and death operates locally within the genome, producing closely-related receptor subfamilies (Figure 2) [5, 37, 42] that may evolve independently of one another. There is evidence that natural selection can operate on the OR repertoire with this finer-scale flexibility to probe niche-specific chemical cues with more precision. For example, Hayden et al. (2010) found that the relative sizes of each mammalian OR subfamily were more representative of a species’ ecotype – aquatic, semi-aquatic, terrestrial, or flying – than of its ancestry within the phylogenetic tree of mammals [52]. These results suggest that particular receptor subfamilies are better suited to probe different environments.

Testing this hypothesis requires identifying the odors sensed by these receptors and determining whether larger subfamilies actually confer a selective advantage in discerning these cues. Observations of subfamily selection on shorter timescales, correlated with even finer niche adaptation, may help hone in on the sensory demands driving receptor evolution. For example, in an analysis of OR evolution in bats, the relative sizes of two subfamilies correlate positively and negatively with independent adaptations to a purely frugivorous diet [66]. A similar investigation of bird and reptile OR repertoires identified a subfamily associated with carnivory and several Class I and Class II subfamilies enlarged in aquatic and terrestrial feeders, respectively [67]. Members of these OR subfamilies may have become tuned to volatile compounds found in the foods sought by individual species, and their expression in the olfactory system may reveal how selective neural pathways adapt to influence foraging and food selection.

Similarly, distinct subfamilies of the mouse V2R tree appear tuned to labile, innate fear-producing chemosignals (“kairomones”) derived from different types of mouse predators (e.g. snakes, birds, rats, and cats) [65]. The observation that the family of Major Urinary Proteins (MUP) acts through V2Rs to control both interspecies defensive behaviors and intraspecies aggressive behaviors suggests that V2R-MUP pairing or binding affinities may be particular targets of lineage-specific subfamily expansion and selection [68–70]. Furthermore, if different V2R subfamilies are in fact tuned to the chemical signatures of particular predators, then their sensory signals could remain segregated as they propagate through the nervous system – which might allow unique defensive responses to evolve for each.

Finally, the modular nature of receptor gene evolution allows small subfamilies to retain critical chemosensory roles, even against a backdrop of widespread receptor loss. One peculiar mammalian subfamily, called “OR37”, is surprisingly conserved in both mice and humans, even though humans have lost more than half of their functional OR genes overall (Figure 2) [50, 71]. Indeed, this subfamily has expanded independently in the human lineage, suggesting it has not lost its utility like other human subfamilies (Figure 2) [72]. Together these data suggest that humans may still detect a set of informative odors through the OR37 family, such as the long-chain fatty aldehyde ligands sensed by mouse OR37 members [73]. Intriguingly, mouse mitral cell projections from OR37 glomeruli target several hypothalamic nuclei rather than the cortical regions typical of most main olfactory bulb projections (Figure 5) [74, 75]. Thus the OR37 family may have evolved to link the sensation of a certain chemical class to regulation of innate behaviors or internal state in a way that has remained crucial across species. The overall pattern of birth-and-death OR evolution therefore demonstrates lineage-specific adaptation to the chemical environment.

Figure 5.

Spatial and molecular organization of projection targets and behavioral responses downstream of distinct mouse olfactory bulb glomeruli. Circumscribed zones of glomeruli are innervated by sensory neurons expressing each of the four largest GPCR OR families or the MS4As. In addition, Class I and Class II OR glomeruli occupy the dorsal-most and more ventral zones, respectively; and projections from Grueneberg Ganglion sensory neurons form their own “necklace” (light blue) anterior to the GC-D/Ms4a “necklace” (green). Dorsal Class II OR glomeruli form a molecularly defined zone, which includes glomeruli necessary for innate aversion of some predator odorants. Within the ventral zone of Class II OR glomeruli, some regions are enriched for the glomeruli of TrpM5- or OR37 subfamily-expressing sensory neurons, which have second-order projections to the “vomeronasal” amygdala and the hypothalamus, respectively.

The Evolution of Single OR Genes: Direct Links Between Sensation and Action

Adaptation through change in the number of OR genes is only possible if the olfactory nervous system has ways to employ the new receptor genes that emerge as particular subfamilies grow and diversify. Because natural odor blends activate only a subset of receptors [22], expressing newly duplicated and diversified subfamily members may provide a more nuanced neural representation of the regions of chemical space they sense. This could in turn allow finer resolution in discriminating the makeup or concentrations of the odors sensed by the ancestral receptors.

What this model does not explain is how an animal’s interpretation of a given odor may evolve. Many vertebrates exhibit innate patterns of behavior that can be released by precise chemical stimuli, including even some monomolecular odorants outside the context of their natural odor blends (Figure 3) [58, 63, 76, 77]. It is not obvious how evolution could couple these blend constituents to appropriate innate behaviors via changes in olfactory receptor number. Animals may instead interpret these chemosignals through single, precisely- and sensitively-tuned olfactory receptors that tap into hardwired, behavior-evoking neural circuits. This pattern of receptor tuning and neural organization is prominent in insects, perhaps because their olfactory receptor repertoires are much smaller (and likely less redundant) than in vertebrates. The compounds that elicit innate responses in both insects and vertebrates are essential for survival and reproduction – sex pheromones, attractive or aversive food odors, and predator signatures – explaining why olfactory neurons have evolved to detect them with high sensitivity and to couple directly to neural circuits that drive innate behaviors (Figure 3) [78–81]. For instance, the exocrine-secreted peptides ESP1 and ESP22 act exclusively through V2Rp5 and an unknown V2R to promote female sexual receptivity (the “lordosis” posture) and to block male sexual activity toward juveniles below reproductive age, respectively [82, 83]. Similarly phenethylamine found in the urine of carnivores, acts by binding to the receptor TAAR4 to promote avoidance behavior [84, 85]. These individual receptors therefore play non-redundant roles in shaping odor perception and interpretation, suggesting that even in olfactory systems that use hundreds or thousands of receptors, single members of these gene families may evolve outsized ecological significance and privileged neural access to specific central circuits [84].

This model also predicts that, in animal lineages in which the role of olfaction in driving stereotyped reproductive and defensive behaviors has waned, there will be few cases where individual receptor genes are under strong selection pressure. This seems to be borne out in humans, for whom there is almost no evidence for positive selection in the coding sequences of single OR genes since the human-chimpanzee divergence ~7 MYA. It is therefore unlikely that particular OR genes have conferred a selective advantage during human evolution [86].

That said, genetic variation in single human ORs – under selective pressure or not – can produce large differences in odor perception. Detection thresholds and (un)pleasantness ratings of many odors have been linked to common polymorphisms in and around human OR genes, including a receptor coding mutation that alters sensitivity to and valence of the male hormones androstenone and androstenedione [87]; a mutation that accounts for nearly all observed variation in the perception of β-ionone and dramatically alters preference for foods emiting this odor [88]; and a polymorphism closely linked to a cluster of OR genes that tracks with the aversive “soapiness” of cilantro [89]. Furthermore, as in other mammals, gland secretions can in a laboratory setting can bias sexual attraction and nipple-orienting behavior in adult and infant humans, respectively, raising the possibility that we employ unidentified pheromones [90] [91]. If these chemosignals follow the pattern of other social and suckling pheromones, they are likely to include monomolecular odorants acting through high-affinity, specialist receptors [77, 92]. Even if they are not under strong selection, then, it remains possible that single human ORs contribute meaningfully to behavior through unknown mechanisms.

Segregated Olfactory Pathways As Substrates For Innate Olfactory Behaviors

Defined odorants can drive innate responses or preferences through a few receptors, even against a background of hundreds or thousands – many of which may be active simultaneously during typical sensation of complex odor blends [93]. How, then, are particular chemical cues coupled precisely to appropriate odor-driven behaviors? It appears that the olfactory system begins to sort odor information in the peripheral sense organs. Several hardwired, parallel pathways – from sensory neurons to subregions of the olfactory bulb to distinct central brain targets – arose early in the evolution of vertebrates, using distinct receptor families and subfamilies to convey different categories of chemical signal. Although zebrafish, unlike mice, have only a single olfactory epithelium, the primary sensory neurons that express ORs and TAARs are molecularly, morphologically, and anatomically distinct from those that express amino acid-sensing V2Rs; these two cell types (ciliated and microvillar sensory neurons, respectively) are found in different layers of the olfactory epithelium and project to separate regions of the olfactory bulb, presaging the separation of mammalian ciliated neurons (in the MOE) and microvillar neurons (in the VNO) and their projections to distinct main and accessory olfactory bulbs [33, 34]. The zebrafish epithelium also contains a third cell type (crypt neurons) that expresses one of the few zebrafish V1R genes in most cells and projects to a unique glomerulus [29, 32, 94]. In addition, a closely related receptor has recently been found to detect a pheromone that releases egg-laying behavior, reminiscent of the reproductive behaviors regulated by mammalian V1Rs (Figure 3) [95]. This “ancestral” organization of the olfactory periphery therefore suggests that discrete subsystems and the receptors they express are hardwired to yield different behavioral responses to odor (Figure 4 top).

Figure 4.

Peripheral organization of OR family expression across vertebrates. Top: A horizontal section of a quarter of one olfactory rosette from the zebrafish (Danio rerio). Sensory neurons are embedded in a main olfactory epithelium (MOE) with a water-filled lumen. Members of four of the five major GPCR OR families are expressed in different sensory neuron types: TAARs and Class I “classical” ORs [here denoting also fish ORs that belong to neither Class I nor Class II] are expressed in ciliated cells occupying the basal layers of the epithelium; V2Rs in microvillar cells in the middle layer; and at least one V1R in “crypt cells” of the upper layer. Axons of the three cell types project to different, spatially segregated, and stereotyped glomeruli of the zebrafish olfactory bulb. Middle: A coronal section of one half of the main olfactory epithelium and the vomeronasal organ (VNO) of the western clawed frog (Xenopus laevis). An air-filled medial diverticulum (MD) houses neurons expressing Class II “classical” ORs and several V1Rs; a water-filled lateral diverticulum (LD), Class I “classical” ORs, several TAARs, and ancestral V2Rs; the water-filled (VNO), newer members of the expanded V2R family. Bottom: A coronal section of the MOE and VNO of the house mouse (Mus musculus). Receptor family expression is segregated (several exceptions not pictured), with Class I ORs and TAARs expressed in the dorsal MOE, Class II ORs in the ventral MOE (and some in the dorsal MOE, not pictured), V1Rs in the apical VNO, V2Rs in the basal VNO, and FPRs in both VNO layers. Some sensory neurons in the “cul-de-sacs” of the MOE express the receptor guanylate cyclase GC-D and multiple members of a non-GPCR family of four-pass transmembrane chemoreceptor family, Ms4a.

Consistent with this possibility, a major olfactory adaptation along the amphibian and terrestrial branches of the vertebrate tree includes the establishment of segregated neuroanatomical structures that express more recently-derived ORs. Unlike fish, amphibians have evolved a separate vomeronasal organ that detects attractive peptide sex pheromones (specific to amphibians) through unusually large and recently expanded subfamilies of V2Rs, as well as an air-filled MOE compartment the expresses the massively enlarged Class II OR repertoire described above; evolutionarily more ancient V2Rs tuned to single amino acids remain expressed in the water-filled MOE compartment along with V1Rs, Class I ORs, and TAARs [62, 96, 97] (Figure 4 middle). The observation that newer V2R genes have acquired both a novel expression pattern and a novel role in innate reproductive behaviors suggests that odor interpretation depends not only on what ligands are detected, but also on the developmental identities and specialized sensory pathways of the cells that detect them [98].

The segregation of the olfactory periphery has proceeded even further in rodents. Mice, for example, have separate epithelial layers for V1Rs and V2Rs within the VNO (Figure 4 bottom); distinct dorsoventral zones and molecular machinery for the expression of TAARs, Class I ORs, and Class II ORs in the MOE [99]; a group of neurons at the tip of the nose, the Grueneberg ganglion, which is thought to detect alarm pheromones and structurally related predator odorants [100, 101]; a patch of olfactory neurons on the nasal septum (the “septal organ”) that largely express a single broadly-tuned OR and may convey information about breathing [102, 103]; and within cul-de-sacs of the MOE, a set of neurons which are thought to mediate the social acquisition of food preference and which express both the receptor guanylate cyclase GC-D and a recently-described family of non-GPCR olfactory receptors (Figure 4 bottom) [104–106].

The olfactory bulb appears to organize this variety of incoming sensory information into domains larger than a single glomerulus. For instance, the sensory neurons that express class I OR-, class II OR-, TAAR-, or GC-D are developmentally constrained to innervate circumscribed and nonintersecting regions of the olfactory bulb (Figure 5) [99] [107]. Furthermore, there is substantial evidence that activity in spatially and molecularly segregated glomeruli drives different patterns of stereotyped behavior via hardwired projections to central brain regions. First, an “innate aversion” domain for predator odor sensation can be found at the ventral subdivision of the dorsal olfactory bulb, while the dorsal-most Class I-expressing domain is instead required for aversive responses to spoiled food odorants [108]. Next, the ventral olfactory bulb houses the glomeruli of molecularly distinct neurons (expressing components of an alternative, TrpM5-dependent chemotransduction pathway) that preferentially respond to urine-derived pheromones and project to the “vomeronasal amygdala”, as well as the cluster of OR37 glomeruli and their unusual mitral cell projections to the hypothalamus [75, 77, 109, 110]. Finally, the TAAR domain of the olfactory bulb is required for innate behavioral responses to several ecologically important amines, even though these same odorants can be detected by receptors in the main olfactory system (Figure 5) [84]. Recent data also suggest a global organization to the main olfactory bulb, in which glomeruli that reside more rostrally evoke investigatory behaviors, while those that are more caudal evoke aversion [111]. Characterizing the mitral cell projections downstream of specific glomeruli may reveal different capacities to influence a variety of central processing or behavioral centers. This “switchboard” organization could allow each OR gene to adapt its chemosensory tuning according to its domain’s function without altering the circuits downstream of the receptor.

However, the interpretation of an odor is not determined simply by whether it recruits neural activity within a specific glomerular domain. For instance within the TAAR domain, the TAAR4 glomerulus is required for aversion to phenethylamine, but the nearby TAAR5 glomerulus is highly tuned to the attractive mouse pheromone trimethylamine [84, 112]. Thus even closely related receptors and their closely apposed glomeruli — within a similar subdomain of the olfactory bulb – may drive opposite behaviors. Combinatorial effects of multiple (TAAR and non-TAAR) glomeruli may produce the different behavioral responses to high-affinity TAAR4 and TAAR5 ligands. Consistent with this possibility, the effect of TAAR5 activation appears to be inverted in rats, for which trimethylamine is aversive (Figure 3) [112]. However it is also possible that, within the TAAR domain of the bulb, TAAR4 and TAAR5 glomeruli are connected to distinct downstream circuits that drive fundamentally different behaviors. Determining whether the differential effects of TAAR4 and TAAR5 ligands in mice relies upon glomerulus-specific projections to higher brain regions, the coordinated activity of groups of glomeruli, or on circuit activity within the TAAR domain will therefore reveal key principles of how odor representations in the olfactory bulb govern perception and behavior.

The appeal of the “domain” hypothesis for coupling odors to behavior lies in its relative developmental simplicity. In this model the olfactory system could take advantage of developmentally-specific gradients of morphogenetic and axon guidance molecules to build sub-circuits within the olfactory bulb that connect to distinct higher brain regions. How might the olfactory system create a system in which individual glomeruli within a given domain are differentially connected to higher brain regions? One highly speculative possibility is that the olfactory bulb’s output layer of mitral cells is, like the retinal ganglion cell layer of the retina, tiled with molecularly distinct cell types that convey different types of information via their projection or spiking patterns. Hardwired synaptic connections to these mitral cell types might then be biased by the complement of cell surface proteins on OSN axons, which are themselves determined by both their developmental identity and the OR gene stochastically chosen for expression [113]. This sort of coupling between ORs and central circuits could explain why many odors appear to have a stereotyped attractive or aversive valence in mice and humans, even when the ORs responsible for their detection are evolving neutrally [114, 115]. In other words, behavioral biases could be substrates for adaptation (e.g. through changes in the expression of cell surface proteins given a particular OR choice) but would also necessarily exist by chance. In this model, the multiglomerular domains of the olfactory bulb may determine which downstream targets are available (e.g. medial amygdala from the accessory bulb and cortical amygdala from the main bulb) but leave enough flexibility for receptor-specific behaviors to evolve. The loss of several subsystems in humans may explain why single OR genes are not under strong selective pressure in our lineage, as a highly-tuned receptor would have no specialized circuitry to tap into to generate an adaptive behavioral response.

Concluding Remarks

How evolution sculpts the olfactory system is a response to both the general problem of probing a wide swath of chemical space, as well as the unique sensory challenges facing each animal species. In vertebrates, both dramatic and subtle adaptations to the chemical environment are possible because of the vast, hierarchical structure of the olfactory receptor (OR) gene families. First, rapid growth or pruning of large OR families accommodate major changes in habitat and lifestyle, as with ocean-to-terrestrial shifts and derived primacy of the visual sense. Second, selection on individual OR families or subfamilies permits niche adaptation or conserved chemical communication between conspecifics. Third, genetic variation in single OR genes, among thousands, can alter behaviors driven by precise chemical cues. We speculate that the molecular logic and neuroanatomical architecture of the vertebrate olfactory nervous system, in which newly arising olfactory receptor genes can tap into a number of segregated pathways that influence both innate and learned behaviors, may have favored this dramatic mode of olfactory receptor evolution. In this model OR genes form the marble from which the vertebrate olfactory system is sculpted, specifying the ability both to detect odors and to implement appropriate survival- and reproduction-promoting responses. Peering into this dynamic process reveals a direct link between the evolution of an animal’s genome and the specific tasks solved by its nervous system.