Strong links between genomic and anatomical diversity in both mammalian olfactory chemosensory systems

Abstract

Mammalian olfaction comprises two chemosensory systems: the odorant-detecting main olfactory system (MOS) and the pheromone-detecting vomeronasal system (VNS). Mammals are diverse in their anatomical and genomic emphases on olfactory chemosensation, including the loss or reduction of these systems in some orders. Despite qualitative evidence linking the genomic evolution of the olfactory systems to specific functions and phenotypes, little work has quantitatively tested whether the genomic aspects of the mammalian olfactory chemosensory systems are correlated to anatomical diversity. We show that the genomic and anatomical variation in these systems is tightly linked in both the VNS and the MOS, though the signature of selection is different in each system. Specifically, the MOS appears to vary based on absolute organ and gene family size while the VNS appears to vary according to the relative proportion of functional genes and relative anatomical size and complexity. Furthermore, there is little evidence that these two systems are evolving in a linked fashion. The relationships between genomic and anatomical diversity strongly support a role for natural selection in shaping both the anatomical and genomic evolution of the olfactory chemosensory systems in mammals.

1. Introduction

Olfaction is a key sensory adaptation of mammals [1]. Colloquially referred to as the ‘sense of smell’, mammalian olfaction actually derives from two chemosensory systems that are distinct at the anatomical, genomic and neurological levels—the main olfactory system (MOS) and vomeronasal system (VNS) (figure 1). Mammals are diverse in their anatomical and genomic development of these two olfactory chemosensory systems, ranging from ‘microsmatic’ cetaceans to ‘macrosmatic’ rodents and canines [3–5].

Figure 1.

Schematic diagram of the anatomical components of the mammalian olfactory chemosensory systems. The diagram shows a line drawing of Microcebus murinus with a CT scan (AMNH 174535) of the nasal fossa in the sagittal plane. The red arrows indicate the length of the VNG, which forms on the palatal portion of the maxilla from the articulation of the cartilaginous capsule surrounding the VNO. When data on VNO length were not available, we measured the VNG length as an osteological proxy. The yellow bracket indicates the perforated portion of the ethmoid bone, through which nerves projecting from the MOE pass and then connect to the MOB. We measured ethmoid area as a proxy for MOE area when data were not available in the literature. Line drawing adapted from Smith et al. [2]. (Online version in colour.)

The MOS detects volatile stimuli using olfactory sensory neurons expressed in the olfactory epithelium (OE) of the nasal mucosa. Olfactory sensory neurons are encoded by the large and genomically scattered olfactory receptor (OR) superfamily [4–7] and expressed in the main OE (MOE). These nerves pass through the cribiform plate of the ethmoid bone and connect to the brain's main olfactory bulb (MOB) yielding information relating to diet, activity pattern, habitat, sociosexual signals and potentially navigation [4,8,9]. On the other hand, the VNS is specialized for the detection of non-volatile stimuli that may play a role in innate responses to odorants responsible for sociosexual behaviours and predator avoidance [10–12]. The peripheral sensory organ of the VNS is the vomeronasal organ (VNO), which comprises bilateral epithelial tubes situated at the anterior base of the nasal cavity [10]. In most mammals, vomeronasal sensory neurons are expressed in the VNO and their axons project to a distinct brain structure, the accessory olfactory bulb. Vomeronasal receptors are encoded by two gene families (V1R and V2R) that are distributed across most chromosomes [13].

Despite considerable variation in the olfactory chemosensory systems of mammals [3–5], little work has explored the relationship between the genomic and anatomical correlates of each olfactory system. In other words, do mammals with well-developed olfactory chemosensory anatomy have relatively well-developed receptor gene families? Hildebrand & Shepherd [14] suggested that receptor cell density (relating to epithelial distribution) would be greater in taxa with larger numbers of olfactory receptor genes. In addition, a positive relationship between the anatomic and genomic diversity has been hypothesized for both the MOS [15] and the VNS [16]. Qualitative studies support a relationship between VNO complexity and VNS gene family evolution [3,17–20]. In addition, quantitative studies of anatomy in birds [21] and ecological behaviour in mammals [4] have shown clear links between genomic and phenotypic evolution of the olfactory chemosensory systems.

Along the lines of these earlier studies, we predict that mammals with larger chemosensory gene family repertoires will have larger and more complex peripheral chemosensory organs. Conversely, in mammals with smaller chemosensory gene family repertoires, we predict a correspondent decrease in anatomical size and/or complexity of their sensory organs. In this study, we quantitatively test this prediction in a diverse set of mammals. We use phylogenetic least-squares regression to test whether anatomical features of the MOS and VNS, such as anatomical complexity and relative and absolute size of sensory organs, are related to features of the chemoreceptor gene families, such as the absolute and relative numbers of functional receptor genes. We interpret any correlation among these variables as evidence for natural selection's role in shaping the evolution of both anatomy and underlying genetics of these olfactory chemosensory systems.

In addition to the relationships within the MOS and VNS, there is considerable debate on the relationship between these two olfactory chemosensory systems. The dual olfactory hypothesis states that each system is adapted to detect specific types of stimuli, namely that the MOS is adapted to detect a broad range of ecological cues while the VNS is specifically adapted to detect pheromones [10,22–25]. Yet, other studies suggest that there is substantial functional overlap between the MOS and the VNS, and particularly that the VNS is not uniquely adapted for pheromone detection [26–29]. To address these questions, we use phylogenetic least-squares regression to test whether anatomical and genomic features of the MOS and VNS are correlated with one another.

2. Material and methods

(a). Anatomical data

We characterized anatomical diversity of the olfactory subsystems for 32 mammals, following the sampling of the genomic analysis of Young et al. [18]. We collected data on the VNO and the surface area of the MOE (the peripheral sensory organ of main olfaction) (figure 1). Owing to the destructive nature of histological methods, data on the sensory epithelium of the VNO or MOE have not been published for all taxa. When possible, epithelial structure of the VNO was recorded from the literature documenting histology and categorized using criteria from Takami [30] who described the VNO of a wide range of tetrapods and Smith et al. [31] who described it in primates (table 1). In this analysis, VNO category was treated as a continuous variable given that each character state represents decreasing complexity on a continuum rather than a discrete morphological trait. Length of the VNO was also recorded when presented in the literature. When these data were unavailable, we used an osteological proxy to estimate VNO length in cranial specimens, the vomeronasal groove (VNG) [32]. We used ethmoid area as a proxy for MOE surface area, following Pihlström et al. [33]. Ethmoid and skull areas (to correct for size) were recorded from Pihlström et al. [33] where available. Size-adjusted ethmoid area was calculated by dividing ethmoid area by skull area. Taxa without published data on VNO length or ethmoid area were CT scanned at the American Museum of Natural History Microimaging Facility in New York, NY. VNG length was collected using ImageJ software [34], and ethmoid and skull area data were extracted from three-dimensional rendered images using Avizo Standard software. Published data on head and body length (HBL) were used to correct for size of the VNO (see electronic supplementary material, table S1 and references therein). Size-adjusted VNO length was calculated by dividing VNO length by HBL. The anatomical variables include VNO category, VNO length (mm) and ethmoid area (mm²). We used both size-corrected and non-corrected values in the analysis and implemented the natural log transformation for non-corrected anatomical variables. When calculating the natural log transformation for VNO length, we added one to our variable prior to transformation so all values were positive.

Table 1.

VNO categories. Six defined character states of the mammalian VNO based on descriptions of epithelial distribution in the literature. VNNE, vomeronasal neuroepithelium; VNS, vomeronasal system; RFE, receptor free epithelium; VNO, vomeronasal organ.

VNO category	description	reference
1	well-developed VNNE, connective tissue components penetrate VNNE, segregated VNS	[30]
2	well developed: VNNE medially with thin lateral RFE	[31]
3	VNNE only	[31]
4	VNNE interrupted by RFE	[31]
5	respiratory-like epithelium, VNO displaced superiorly	[31]
6	absent	[31]

(b). Gene family data

We compiled data on the olfactory genomes by recoding published values reported in Hayden et al. [4] and Matsui et al. [35] for the OR gene family, and Young et al. [18] for the V1R gene family. From these data, we calculated the total number of genes, the total number of presumed functional/intact genes and the proportion of functional/intact genes for both gene families (electronic supplementary material, table S1). A second class of vomeronasal receptor genes, V2R, was not included in the analysis because functional V2R genes have only been identified in a small number of mammals [17,36].

(c). Quantitative methods

We tested for relationships within and between the anatomical and genetic components of the MOS and VNS using phylogenetically corrected generalized least-squares (PGLS) regression using the caper package [37] of R [38]. We used a pruned version of the mammalian supertree of Bininda-Emonds et al. [39] for all analyses. The integrated likelihood function of caper was used to estimate the phylogenetic scaling parameter λ [40]. To correct for Type I error when performing multiple tests, we used Benjamini & Yekutieli's [41] false discovery rate approach, as well as the Bonferroni correction to rescale significance levels (calculated in R using p.adjust) [38].

3. Results

Data for multiple variables were collected or compiled for 32 different species of mammals (electronic supplementary material, table S1). From these data, 11 variables were extracted for analysis. For the MOS, we analysed three genomic features of the mammalian OR superfamily: the total number of OR genes, the number of functional OR genes and the proportion of functional OR genes. As an anatomical index of the MOS, we examined the absolute and relative area of the cribiform plate of the ethmoid bone (referred to as ethmoid area), a predictor of the size and performance of the MOS [33].

Phylogenetically corrected regressions show that absolute ethmoid area is positively correlated with both the total and functional number of OR genes (table 2 and figure 2a; electronic supplementary material, table S2 and figure S1). There is no relationship between the proportion of functional OR genes and any other variable, though smaller bodied mammals (e.g. rodents) appear to have higher proportions of functional OR genes when compared with larger mammals (figure 2a).

Table 2.

Pairwise comparisons of variables using PGLS regression. The top left quadrant denotes VNS-related variables and comparisons; bottom right denotes MOS and top right/bottom left denotes comparisons between the VNS and MOS. Above diagonal: probability values for each of the pairwise PGLS regressions. Bold p-values are those significant after controlling for the false discovery rate using the method of Benjamini & Yekutieli [41]. Italic p-values are those significant after both the Bonferroni correction and the false discovery method. Below diagonal: R² values for each pairwise comparison.

Figure 2.

Relationships among selected VNS and MOS variables. Panel (a) shows the total number of OR genes on the y-axis and absolute ethmoid area on the x-axis. The size of each point reflects the proportion of functional OR genes, which is not significantly related to ethmoid area. The colour of each point indicates the overall size of each particular mammal (log head and body length). Panel (b) shows size-adjusted VNO length on the x-axis and the proportion of intact V1R genes on the y-axis. The size of each point reflects the morphological complexity of the VNO, which is also positively correlated with both size-adjusted VNO length and the proportion of intact V1R genes. The colour of each point indicates the overall size of each particular mammal (log head and body length). Superimposed points were ‘jittered’ for visualization purposes; inset plots show points in their original position. Scatterplots of all comparisons appear in electronic supplementary material, figures S1–S3 and all regression statistics appear in electronic supplementary material, table S2.

For the VNS, we examined three genomic features of the main family of receptor genes (V1R): the total number of V1R genes, the number of intact V1R genes and the proportion of intact V1R genes. We examined three of anatomical variables of the VNO: its overall complexity, as well as its absolute and relative length.

Within the VNS, the proportion of intact V1R genes is positively correlated with both the absolute and relative length of the VNO and its anatomical complexity (table 2 and figure 2b; electronic supplementary material, table S2 and figure S2) based on phylogenetically corrected regressions. Additionally, VNO complexity scales with absolute and relative VNO length. As in the MOS, there are highly significant positive correlations among variables, suggesting that natural selection has shaped the genomic and anatomical variation of the VNS together.

There are few significant relationships between the MOS and VNS, and those that exist could be explained by changes in body size (table 2; electronic supplementary material, table S2 and figure S3). Significant relationships exist between VNO length and several variables within the MOS, including ethmoid area, the total number of OR genes and the number of functional OR genes. These relationships likely reflect the relationship between overall size and the MOS (see Discussion), given the significant relationship between VNO length and body length (p-value < 0.01) (electronic supplementary material, figure S4).

4. Discussion

Within each olfactory chemosensory system, there are features of chemosensory anatomy that are strongly related to specific aspects of their constituent gene families. Interestingly, these relationships differ in the MOS and VNS. The VNS is modulated in proportional terms—the proportion of intact V1R genes relates to multiple aspects of VNO anatomy, including its relative size. This is supported by the large proportion of variance explained by the linear relationship between percentage of intact V1R genes and the relative size of the VNO (R² = 0.70), which was comparatively larger than all other coefficients of determination among the VNS variables (table 2; electronic supplementary material, table S1). On the other hand, the MOS responds in absolute terms, both in anatomical size and numbers of OR receptors. Indeed, functional studies show that the absolute size of the olfactory organ is closely related to olfactory sensitivity, rather than its relative size [33]. The pattern is somewhat similar in birds, where the relative olfactory bulb size is related to the absolute number of OR genes [21]. Both our findings and those of Steiger et al. [21] support a link between absolute numbers of OR genes and olfactory ability [42].

Furthermore, there are fewer significant relationships between the MOS and VNS, except for a relationship between VNO length and ethmoid area, the total number of OR genes and the number of functional OR genes. These likely reflect the relationship between overall body size and the MOS, an interpretation in line with observations by Pihlström et al. [33] that the MOS is not under size constraints that might limit other sensory organs. In the VNS, on the other hand, we find that relative size of the VNO and relative number of intact V1R genes is positively correlated suggesting that the VNS is limited by size constraints. Neither sensory neuron bodies nor cell density scales isometrically with body size, therefore absolutely larger surface areas for olfactory sensory neurons have a positive relationship with olfactory sensitivity [33,43]. Alternatively, the positive relationship between VNO length and OR gene repertoires could be the result of both olfactory chemosensory systems evolving to be more complex in taxa that are more reliant on both systems. Additionally, selection for larger surface areas for main olfactory and vomeronasal neuroepithelium could result in larger nasal cavity size, potentially driving absolute sizes of the ethmoid area and VNO together.

We interpret these findings as evidence that natural selection has worked to shape the genomic and anatomical variation of both chemosensory systems, though selection may not act on them in a tightly correlated fashion. The MOS may respond to selective pressure in absolute terms, i.e. in absolute size and total number of functional gene copies. Indeed, it may be constrained to respond in an absolute size-dependent manner [33]. The VNS appears to respond to selection by varying relative size and complexity of its constituent features. Mammals that are more reliant on vomeronasal olfaction maintain larger proportions of functional V1R genes in a relatively longer and more complex VNO. Mammals that are less reliant on the VNS are under reduced selection to maintain complex VNOs and functional genes, thereby allowing for the accumulation of pseudogenes. This would account for lower percentages of functional genes in species with a less complex and relatively smaller VNO. These differences in the evolutionary patterns of the MOS and VNS can be interpreted to extend the differential tuning hypothesis [44], which states that each olfactory chemosensory system is under a distinct selective regime where the MOS is broadly tuned to detect a wide range of odorants, while the VNS is more finely tuned and under lineage-specific selection. Our results suggest that the differential tuning hypothesis [44] can be extended to include differences in the specific anatomical and genomic responses of the VNS and MOS to selective pressures. While our results do not bear on the distinct functions of each olfactory system, they support increasing evidence for the role of the VNS in innate responses to sociosexual and predator information, while the MOS may respond to a broader set of stimuli including some overlap in detecting sociosexual and predator odorants [11].

One noteworthy caveat is apparent when considering the MOS of the smallest and largest mammals. In figure 2a, the mouse (red) has a very high proportion of functional OR genes, yet its absolute number of functional OR genes is unremarkable. The elephant (blue) has an exceptionally large absolute number of functional OR genes, yet the proportion of functional genes is modest. This may be interpreted as two size-specific strategies. Smaller mammals may increase their proportion of functional OR genes as a size-limited ‘macrosmatic’ strategy constrained by their smaller available surface area for sensory neurons. Additionally, the significant relationship between relative VNO length and the number of functional OR genes reflect an enhanced olfactory ability in the largest mammals. As such, larger mammals may have greater capacity to pursue an overall ‘macrosmatic’ sensory strategy. Additional work is needed to determine how size affects the different olfactory chemosensory systems and sensory strategies of mammals, especially in the largest and smallest species.

These results also bear on the debate over the relative contributions of adaptive and random processes in the evolution of the MOS and VNS receptor gene families [45]. Interestingly, some genetic analyses provide evidence for adaptive evolution within the OR and V1R gene families [46–51], while others suggest that random processes also play a role [19,45,52–54]. Hayden et al. [4] recently argued for an adaptive interpretation of OR gene family evolution based on relationships between OR types and ecology in a wide array of mammalian species. Our analyses are similarly supportive of an adaptive interpretation for both the MOS and VNS, given the close relationship among multiple genomic and anatomical variables across mammals. Taken together, there is evidence for adaptive evolution in the mammalian olfactory chemosensory systems from evidence based on gene sequences [46,51], receptor copy numbers [18,52], anatomical diversity [18] and ecological parameters [4]. A challenge remains in reconciling the compelling evidence for neutral interpretations of olfactory chemoreceptor gene family evolution [19,45,52–54] with these multiple levels of evidence for the adaptive evolution in the olfactory chemosensory systems of mammals.

Funding statement

This research was supported by NSF grant nos. 0961964 (to E.G.) and 0966166 (NYCEP IGERT). The infrastructure of the Anthropological Genetics Laboratory at Hunter College was supported, in part, by grant no. RR003037 from the National Center for Research Resources, a component of the National Institutes of Health, its contents are solely the responsibility of the authors and do not necessarily represent the official views of NCRR, NIMHD or NIH.