The Role of Ortho-Retronasal Olfaction in Mammalian Cortical Evolution

Abstract

Fossils of mammals and their extinct relatives among cynodonts give evidence of correlated transformations affecting olfaction as well as mastication, head movement, and ventilation, and suggest evolutionary coupling of these seemingly separate anatomical regions into a larger integrated system of ortho-retronasal olfaction. Evidence from paleontology and physiology suggests that ortho-retronasal olfaction played a critical role at three stages of mammalian cortical evolution: early mammalian brain development was driven in part by ortho-nasal olfaction; the bauplan for neocortex had higher-level association functions derived from olfactory cortex; and human cortical evolution was enhanced by ortho-retronasal smell.

Graphical Abstract

Evidence from paleontology and physiology suggests that ortho-retronasal olfaction played a critical role at three stages of mammalian cortical evolution: early brain development was driven partly by ortho-retronasal olfaction; the bauplan for neocortex had higher-level association functions derived from olfactory cortex; and human cortical evolution was enhanced by ortho-retronasal olfaction.

Introduction

In analyzing mechanisms in human brain evolution, vision is usually considered paramount and olfaction of minor importance. Moreover, across all primates an evolutionary trade-off in neural processing volumes and performance has been hypothesized between specializations in the visual and olfactory systems. Primates with high visual performance are thought to have small olfactory processing centers and correspondingly diminished olfactory performance, and vice versa (Barton et al., 1995). However, olfaction is a dominant sense in the behavior of most mammals (Stoddart, 1980) and moreover, the convergence of orthonasal and retronasal signals (Rozin, 1982) lies in neocortical areas that are tied to human cravings responsible for disorders such as obesity and food addiction (Shepherd, 2012). Yet the evolutionary history of this unique duality of orthonasal and retronasal olfaction has yet to be studied in detail.

Here we reconsider the role of ortho-retronasal olfaction in mammalian cortical evolution in the light of recent developments in paleontology and cortical physiology. In integrating observations from these traditionally separate fields we will provide evidence for the role of ortho-retronasal olfaction at three critical stages in cortical evolution: the earliest pulse of pre- mammalian encephalization; the transition from three layer to multilayer cortex; and the enlargement of neocortex in Homo sapiens.

Defining the Enlarged Sense of Ortho-Retronasal Olfaction

Mammals generally retain the primitive tetrapod olfaction mode of sniffing known as ‘orthonasal’ smell, in which airborne environmental odor molecules are drawn through the nares (nostrils) into the nose to activate the olfactory epithelium. Orthonasal smell in mammals has its own special characteristics which derive from a huge olfactory receptor (OR) genome. Approximately ~1200 OR genes are thought to have been present in mammals ancestrally, compared to ~100 that were present in the first tetrapods and amniotes (Niimura, 2009, 2012). Mammals also employ a system of diaphragmatic ventilation that, together with other distinctive features such as their modes of head movement, confers unique attributes to mammalian orthonasal smell, such as their abilities in scent-tracking.

The counterpart to orthonasal smell is ‘retronasal’ smell, in which air exhaled from the lungs carries with it an entirely new information domain of odor molecules liberated in the mouth through the breakdown of food by chewing, saliva, and actions of the tongue (Fig. 1). These molecules pass forward from the caudal part of the mouth via the choana (internal naris) and across the main olfactory epithelium before being expelled through the nares. In retronasal smell, olfaction combines with taste and other senses (e.g., somatosensation, vision, hearing) to generate our sensation of flavor (Shepherd, 2004, 2006, 2012). Orthonasal smell, retronasal smell, taste, and somatosensory signals from the lips, tongue and teeth all converge in the neocortical area known as the orbitofrontal cortex (De Araujo et al., 2003; Small et al, 2007; Rolls and Grabenhorst, 2008). That flavor is a multisensory map in which distinct classes of information are integrated is evident in clinical data from patients who lost olfactory sensation following nasal infection or cranial trauma (Cullen and Leopold 1999; Franselli, et al., 2004; Bonfils et al., 2005) and from laboratory experiments (e.g., Heilmann and Humel, 2004; Sun and Halpern, 2005; Gautam and Verhagen, 2012).

Figure 1.

Comparison of orthonasal and retronasal smell in dogs and humans, showing the human adaptations for retronasal smell (from Shepherd, 2012).

Retronasal smell and the ortho-retronasal duality in the construction of flavor are unique to mammals among living species, as we explain below. Flavor is cognitively experienced and referred to the mouth. To emphasize the new appreciation of the duality of mammalian olfaction, involving both external smells and internally-generated volatiles, we will refer to olfaction with the full term: ‘ortho-retronasal olfaction’. Many facets of ortho-retronasal olfaction are dependent on the spatial organization and mechanical performance of the skull, dentition, and postcranial skeleton. In this light, paleontology offers special insights into the role of olfaction in cortical evolution.

Ortho-Retronasal Olfaction and the Origins of Mammalian Cortex

As detailed below, the skeletal basis for ortho-retronasal olfaction can be traced in the fossil record back to the earliest members of Cynodontia (Fig. 2). Older literature casts Cynodontia as an extinct group, but used here in its monophyletic sense the name also encompasses ‘crown’ Mammalia, that is, the taxon stemming from the last common ancestor of living monotreme and therian species and all its descendants (Rowe, 1988). Cynodontia includes all taxa descended from the last common ancestor of mammals and the Late Permian fossil Procynosuchus delaharpae (Rowe, 1993). Cynodontia is a member of the more inclusive clade Synapsida, which diverged from other amniotes by the Late Carboniferous, ~310 million years ago (Ma). The first cynodonts appeared ~ 260 Ma (Gauthier et al., 1988), and crown Mammalia originated by the Early or Middle Jurassic, ~180 Ma (Rowe, 1988).

Figure 2.

Outline of phylogeny of Synapsida, showing some of the main features of skeletal evolution related to ortho-retronasal olfaction (modified from Dingus and Rowe, 1998)

In recent decades, a succession of phylogenetic analyses (see Methods) identified well corroborated elements of cynodont phylogeny while also mapping character variation among the skeletons of early cynodonts and mammals. Correlated historical patterns of variation in the braincase, dentition, skeleton, and in endocasts of the brain, combined with developmental and experimental data, offer new evidence suggesting a driving role for ortho-retronasal olfaction in shaping cynodont and cortical evolution in the origins of mammals and humans.

Materials and Methods

Phylogenetic analyses

Our review of morphological evolution is based on a quarter-century of phylogenetic analyses aimed at understanding the relationships of mammals and their extinct relatives (Gauthier et al., 1988; Rowe, 1988, 1993; Luo and Wible 2005; Kielan Jaworowska et al., 2004; Ji et al., 2006; Meng et al., 2006; Rougier et al., 2007; O’Leary et al., 2013). These analyses were largely free of functional or mechanistic agendas, and were conducted in ever-increasingly detailed comparative analyses of the distribution of variable skeletal and dental characters, with the assistance of rapidly evolving computational algorithms for parsimony, maximum likelihood, and Bayesian analysis (Swofford, 1998, 2003; Ronquist and Huelsenbeck, 2003). Although differing in details, a stable pattern of relationships and character distributions among major clades of mammals has emerged and forms the basis of our review. Methods for measuring encephalization quotient (EQ) estimates cited below are described in Rowe et al. (2011).

Developmental Analyses

In addition to literature cited, our observations on ontogeny are based on a growth series of specimens of the marsupial Monodelphis domestica, of precisely documented ages provided by the Southwestern Foundation for Biomedical Research, and fixed under protocols approved at the time (VandeBerg, 1990). The collection includes serial-sectioned histological preparations (azocarmine) of five specimens (postnatal days 0, 10, 15, 26, and 36), 120 cleared and double-stained whole preparations (aged from day 0 through retired breeders), and 30 dried skeletons representing different ages. The histological serial sections were studied to trace development of the olfactory system and to provide landmarks for interpreting the distributions of different epithelial types in imaging analyses.

Imaging Analyses

Using high-resolution X-ray computed tomography (CT), 31 specimens of Monodelphis, and nearly 1000 recent and fossil mammals have been CT scanned and archived at the University of Texas High-Resolution X-ray Computed Tomography Facility since it began operation in 1997 (Rowe et al., 1997; Carlson et al., 2003; Rowe and Frank, 2011). This collection forms much of the comparative anatomical basis for this study, in addition to literature cited. The scanned Monodelphis specimens ranged from 1-day-old specimens through retired breeders, and included dried skeletal preparations, ETOH preserved, and frozen whole individuals.

Results

Development of the Ortho-Retronasal Olfactory System and its Skeleton

Olfaction is the first of the mammalian sensory systems to differentiate. Both the main olfactory system (MOS) and vomeronasal system (VNS) develop from a single pair of ectodermal olfactory placodes that form at the rostral extremity of the neural plate (Schlosser, 2010). Soon after gastrulation, they invaginate to contact the rostral end of the neural tube, initiating organogenesis and the formation of their direct synaptic connection to the presumptive telencephalon. The rostral position of the olfactory placodes may explain why the olfactory system is the only sensory system that projects directly to the telencephalon; the other cranial sensory placodes are positioned laterally or caudal to the presumptive diencephalon and form mature pathways to the telencephalon via the thalamus (Schlosser, 2010).

Contact between the placode and neural tube induces the onset of olfactory receptor (OR) and vomeronasal receptor (VR) gene expression. Primary contact by axons from the first ORs and VRs induces differentiation of the olfactory bulb (OB) and the accessory olfactory bulb (AOB), respectively, in the rostral telencephalon. From this moment onwards, the MOS and VNS diverge onto independent ontogenetic trajectories.

Growth of OR cells induces the development of an expansive olfactory epithelium (OE) of the nasal capsule. The VRs mostly consolidate in the vomeronasal organ (VNO), but some are spread diffusely over the nasal septum (Rowe et al., 2005) and possibly intermingled in the main OE, as they are in fish-like vertebrates (Bruce and Braford, 2009).

The vomeronasal organ (VNO) is present in most mammals, but absent in humans and probably reduced to a diffuse epithelium in cetaceans (Colbert et al., 2005). It generally forms a blind cylinder that opens through a duct into the roof of the mouth or onto the floor of the nasopharyngeal passage inside the nares. VRs occupying the epithelium of its lumen are encoded by V1R genes that are sensitive to small volatile molecules, and V2R genes which are sensitive to soluble molecules (Dulac and Torell, 2003). The VNO functions in a range of intraspecific behaviors and plays a role in feeding in some marsupials (Ashwell, 2010). Axons from the VNO make their first synapse in the accessory olfactory bulb (AOB), which in turn projects to the medial amygdala, and to the posterior medial cortical amygdala (Bruce 2007, 2009). Variation in numbers of V1R and V2R genes among mammals suggests variation in VNO performance, but few behavioral correlates are yet known. The VNO is most highly elaborated the platypus, in which the V1R genome is 50% larger than any other vertebrate yet reported, with 270 functional and 579 pseudogenes, and 83 more intact genes than any other mammal (Grus et al., 2007; Shi and Zhang 2007). Its V2R genome has 15 intact and 112 pseudogenes, with 10 of the 15 functional genes segregating into a platypus-specific clade (Grus et al., 2007).

Except for one or more foramina in mammals marking the passage of the Terminal Nerve (CN 0) to the AOB, the VNO itself leaves no obvious anatomical signal in the fossil record summarized below. In cases where the AOB is visible on cranial endocasts, it plays little or no role in the evolution of encephalization (Rowe et al., 2011). The VNS seems ripe for further research to map its anatomical and genetic variation and to understand more fully its behavioral role in different species. Despite its developmental origin in the olfactory placode, there is currently no clear evidence that the VNS contributes to the multi-sensory map that we refer to broadly as ortho-retronasal olfaction. In the limited space available we therefore focus on the relationship of ortho-retronasal olfaction to the MOS and to skeletal features that are prone to fossilization.

ORs of the MOS are G protein-coupled neurons, and each receptor type is encoded by a separate gene and sensitive to its own narrow class of odor molecules (Buck and Axel, 1991; Hildebrand and Shepherd, 1997; Mombaerts 2001, 2004; Bargmann, 2006). In all mammals studied to date, each OR neuron expresses only one of a possible ~ 1200 ORs encoded in the OR genome (Komiyama and Luo, 2006). Mammalian ORs tend to have a zonal distribution pattern in the OE (Ressler et al., 1994; Vassar et al., 1994). ORs also have short life spans of only about 60 days, and they are continuously replaced over the lifespan of an individual from populations of olfactory stem cells maintained in the OE. As new OR cells differentiate to replace senescent cells they presumably express the same OR gene as their predecessors, although the mechanism for gene specification is unknown.

The OE begins its development on the inner walls of the cartilaginous embryonic nasal capsule. In crown Mammalia (but not its extinct relatives), the nasal capsule becomes extensively ossified and within it grows an elaborate labyrinth of thin bony struts known as ‘ethmoid turbinals’ (or turbinates). As OE growth quickly exceeds the surface area of the nasal capsule walls, it folds into the lumen of the capsule (Fig. 3). Each OE fold is supported by a transient cartilage that grows apically into the fold from the nasal capsule wall, but at no time is there an extensive, stand-alone cartilaginous framework. The growing cartilage is quickly replaced by rigid perichondral bone that forms the mature ethmoid turbinals. Growth of the OE and its turbinals begins adjacent to the OB and proceeds rostrally. As they grow, the turbinals widen rostrally, branching and interleaving in intricate patterns that eventually occupy a large volume of nasal space. The mature OE is confined to the dorsal and caudal regions of the nasal chamber, where the turbinals sequester numerous pockets and recesses into which odorant molecules volatilize. The turbinals subdivide the nasal chamber, maintain spatial integrity of its epithelia, and the spatial zonation of ORs. The number of functional OR genes correlates most strongly with mature OE surface area (Garrett and Steiper 2014). The ossification of ethmoid turbinals in the ancestral mammal expanded the surface area of its olfactory epithelium by an order of magnitude over nasal chambers lacking such structures (Rowe et al., 2005).

Figure 3.

Histological sections through Monodelphis at day 10 (A, B) and day 36 (C, D) showing development of respiratory and olfactory turbinals. Sections A and C are anterior to the orbit, through the nasal capsule. Sections B and D are through the orbit, and show the transverse lamina separating the blind sphenethmoid recess from the nasopharyngeal passage just anterior to the choana. Abbreviations: EtT, ethmoid turbinal, Me, mesethmoid; NPP, nasopharyngeal passageway; OB, olfactory bulb; OE, olfactory epithelum; SeR, sphenethmoid recess; SP, secondary palate; TvsL, transverse lamina; VNO, vomeronasal organ; Vo, vomer. Image brightness and background uniformity were adjusted in Adobe Photoshop.

While the walls and floor of the nasal capsule tend to ossify, two major penetrations allow airflow through what is known as the ethmoid recess. Rostrally is the fenestra narina which becomes the naris. Penetrating the floor is the fenestra olfactorium, which becomes the choana. Primitively, the choanae communicate to the front of the oral cavity. In mammals with a secondary palate (below) the fenestra olfactorium opens into the nasopharyngeal passageway, which displaces the choana to the back of the oral chamber. The nasal capsule floor in front and behind the fenestra olfactorium may ossify as the anterior and posterior transverse laminae, and when present in mammals they form a partial roof over the nasopharyngeal passageway. One or both fail to develop in some mammal clades (below), and other penetrations such as sphenethmoid apertures are known (Fig. 4). The architecture of the turbinals and floor of the nasal capsule determines whether orthonasal and/or retronasal air currents pass directly over olfactory epithelium. Turbinals are highly variable between mammalian clades, and only recently has 3D imaging made morphological analysis possible (Van Valkenburgh et al., 2004; Rowe et al., 2005; Smith et al., 2011; Eiting et al., 2015), and shown that they preserve phylogenetic (Macrini, 2012) and ecological (Van Valkenburgh et al., 2011) signals.

Figure 4.

Mature skull of Monodelphis reconstructed from CT data. A) dorsal view cut-away to show cribriform plate (yellow); (B) ventral view, with jaws (blue) and secondary palate (red) with arrows showing retronasal entrance to the nose via the choanae; (C) jaws and part of secondary palate removed with arrows showing the sphenethmoidal apertures in the ossified floor of the nasal capsule (yellow), which direct retronasal airflow across olfactory epithelium.

Ossified respiratory turbinals covered by mucociliary epithelium develop in a similar fashion within the rostral portion of the nasal capsule, although little is known of their genetic basis. They lie directly in the stream of ortho-retronasal airflow and function in metabolic heat and water balance (Smith et al., 2011; Eiting et al., 2015). Saturation of air within the nose may also play a role in volatilization, but experimental airflow studies are still in their infancy.

The number of functional OR genes determines the number of glomeruli, which are induced as the OR axons enter the presumptive olfactory bulb to make their first synapse (Farbman, 1988, 1990; Mombaerts 2001; Chen and Shepherd, 2005; Bargmann, 2006). Each glomerulus is receptive to a particular type of OR, and axons from each OR type generally converge on only two glomeruli in the main OB, for a convergence ratio of 1 (OR type) to 2 (glomeruli). Humans are the only known exception to this last rule, recently being shown to have >5500 glomeruli even though they express only ~350 functional OR genes (Maresh et al., 2008). The human convergence ratio of 1:16 suggests that generation and recruitment of additional glomeruli confers a unique measure of olfactory performance despite the evolutionary reduction in number of functional OR genes, but little more can be said at present.

An ontogenetic interdependency plays out as OR gene expression drives multiplication of ORs and their supporting cells, in turn driving growth of the OE and its bony turbinals. Equivocal evidence from the fossil record (below) hints at partial ossification of nasal capsule elements prior to the origin of mammals (Kielan-Jaworowska et al., 2004; Ruf et al., 2014). However, full ossification of the ethmoid and its turbinals arose in the last common ancestor of crown Mammalia (Rowe, 1988), potentially accommodating expression of the full complement of ~1200 OR genes believed present ancestrally (Rowe et al., 2011).

With a ten-fold increase in OR genes and ORs, the problem of axonal guidance to their proper glomeruli was amplified proportionately. 3D imaging suggests that the shape of each turbinal is essentially that of a funnel, which grows from the foramina of the cribriform plate rostrally towards its mature, wide-mouthed terminus (Rowe et al., 2005). Early developmental studies (Harrison, 1910) discovered that axonal guidance was determined by the geometry of their physical substrate, and that they grow in a straight line along solid substrates or follow visible topographic paths when such paths were visible. Axons were never observed to shift trajectories unless another solid surface or interface was available (Harrison, 1914). The funnel-shape of each turbinal provides passive, longitudinal guidance as the growing OR axons are funneled very close to their target glomerulus, just across the cribriform plate (Rowe et al., 2005). A host of molecular guidance factors have also been proposed (Singer et al., 1995; Wang et al., 1997; Bozza et al., 2002; Vassalli, et al., 2002; Mombaerts 2004, 2006), and how these complement the turbinal architecture remains to be determined.

Primary projections from the olfactory bulb (Fig. 5) pass via the lateral olfactory tract to the pyriform (olfactory) cortex, with fibers reaching also the anterior olfactory nucleus, olfactory tubercle, entorhinal cortex, and several nuclei in cortical amygdala (Nieuwenhuys et al., 1998). Odors are believed to be encoded by differential activation of the glomeruli to form ‘odor images’ that are transformed by the olfactory cortex into ‘odor objects’ as the neural basis for odor discrimination (Shepherd, 1991, 2013; Wilson and Stevenson, 2006). These connections have been observed in mammals (Ashwell, 2010, 2013), turtles and squamates (Bruce, 2007, 2009) and can be inferred as present in amniotes ancestrally. However, the structural basis for a dual ortho-retronasal olfactory system is unique to Mammalia and its extinct relatives among Cynodontia (below).

Figure 5.

Circuitry schematic of modern opossum (Didelphis) brain showing A) sensory inputs and B) motor outputs (From Rowe et al., 2011).

Humans follow this general developmental pattern, although in a truncated form. Only about ~350–400 OR genes are expressed, and human turbinals and OE are correspondingly reduced from the condition inferred to have been present in mammals, therians, and primates ancestrally. Additionally, the floor of the human nasal capsule fails to ossify and neither anterior nor posterior transverse lamina forms (Smith and Rossi, 2006), leaving the human olfactory epithelium broadly open to orthonasal and retronasal air currents.

Early Evolution of Ortho-Retronasal Olfaction

Orthonasal olfaction, that is, sniffing in, is an innovation of Tetrapoda, and its history can be traced back into the fossil record to the earliest Devonian fossils belonging to the tetrapod stem (Jarvik, 1942). Known as rhipidistian crossopterygians, these transitional stem-tetrapods were the first vertebrates in which the naris conveyed environmental molecules across the OE and into the mouth via the choana. Orthonasal smell is employed by nearly all tetrapod species, the exceptions being secondarily adapted to a committed aquatic lifestyle such as odontocete cetaceans (Colbert et al., 2005; Racicot and Rowe, 2014) and sirenians, in which the main olfactory system is largely or wholly abandoned, and possibly the terrestrial lungless plethodontid salamanders.

With the origin of Amniota, orthonasal airflow became tied to two distinct functions, each supported by a primary ‘choncha’ or epithelial fold that covered a low ridge of cartilage protruding into the lumen from the lateral wall of the nasal capsule. The anterior choncha consists of mucociliary respiratory epithelium and represents the primordium of the mammalian respiratory turbinal, while the posterior concha comprises olfactory epithelium and represents the primordium of mammalian olfactory turbinals (Gauthier et al., 1988). Building on this foundation of olfactory organization, we next discuss evidence preserved in fossil cynodonts relating to the origin of ortho-retronasal smell and mammalian cortical structure, using the cladogram in figure 6.

Figure 6.

Cynodont phylogeny described in text, with key characters indicated.

Origin of Cynodontia and Retronasal Olfaction

During the first ~50 million years of synapsid history, there is little evidence of olfactory or cortical modification. Basal synapsids were eye-minded terrestrial macropredators that dominated the terrestrial trophic ecosystem throughout the Permian and into the mid-Triassic. Major themes in early synapsid evolution involved increased frontality of the orbits, increased bite forces, modest dietary diversification, and improved speed and agility. These changes surely involved cortical modifications, but the early braincases were only partly ossified and fail to record tangible evidence. Early synapsid braincases were organized much like those of basal amniotes.

The first measurable changes in the brain occurred with the origin of Cynodontia¹ in the Late Permian. The early cynodonts remained terrestrial quadrupeds but were smaller than their Early Permian forebears, being roughly the size of a domestic cat. Their olfactory bulbs remained small and the nasal capsule was entirely unossified. The cerebellum was wider than the forebrain, the midbrain was exposed dorsally, and a small pineal eye occupied a canal that perforated the skull roof. Compared to living mammals, the first cynodonts possessed low-resolution olfaction, insensitive hearing with massive middle ear ossicles still attached to the mandible, coarse tactile sensitivity, and relatively unrefined motor coordination (Rowe et al., 2011). They were scratch diggers, and evidence that they may have hibernated comes from specimens found curled and preserved in burrows (Groenewald et al., 2001). Nevertheless, when compared with more primitive synapsids, early cynodonts record a dietary shift to more general omnivory that is correlated with the first of three successive pulses in encephalization that preceded the origin of Mammalia (Rowe et al., 2011).

First pulse in Encephalization

The lateral wall of the cynodont braincase became ossified by ventral extensions of the frontal and parietal bones, forward expansion of the prootic, and by the newly formed alisphenoid. The alisphenoid is a compound element built from an embryological remnant of the epiperygoid footplate, and a new membranous ossification of the spheno-obturator membrane that consistently lies adjacent to the caudolateral pole of olfactory cortex in mammals (Maier, 1987; Gauthier et al., 1988). Appearance of the alisphenoid correlates with expansion of the olfactory cortex, and this first pulse of brain expansion raised early cynodont encephalization quotient (EQ) to ~ 0.20 (based on an average for living mammals; Rowe et al., 2011). In non-mammalian cynodonts, the alisphenoid is referred to interchangeably as the epipterygoid. However, the former term is preferable because the critical transformation combining two separate elements is characteristic of cynodonts.

The Secondary Palate

One of the most diagnostic features of Cynodontia, and a key structural element in ortho-retronasal olfaction, is the secondary palate. In basal synapsids, the choanae were located at the front of the mouth (Fig. 7). With simple conical teeth, these macropredators used their mouths to subdue and dismember prey, and their mode of inertial swallowing was aided by teeth on the roof of the palate (Kemp, 2005). The appearance of the secondary palate marked a profound reorganization in feeding behavior. Palatal teeth and the inertial mode of swallowing were lost in cynodonts, and the tongue became the major guide of food around the mouth and toward the esophagus (Barghusen, 1986; Crompton, 1989). The uniquely complex chewing and swallowing behaviors of mammals became the basis of retronasal olfaction in early cynodonts.

Figure 7.

Stages in the evolution of mammalian secondary palate and the ortho-retronasal olfaction duality. (A) Eusthenopteron, a stem-tetrapod; (B) Seymouria, a stem amniote; (C) Dimetrodon, a basal synapsid; (D) Syodon, a more derived non-cynodontian synapsid; (E) Procynosuchus, the basal-most cynodont with an incipient secondary palate; (F) Thrinaxodon, an early cynodont with a complete secondary palate; (G) Kayentatherium, a basal mammaliamorph with a complex dentition; (H) Morganucodon, a basal mammaliaform, with secondary palate extending to back of tooth row; (I) Didelphis, with secondary extending behind tooth row. Abbreviations: ch – choana; max – maxilla; pal – palatine; pmx - premaxilla; pt – pterygoid; vo – vomer.

The secondary palate separates the oral cavity from the nasal cavity as shelves of the maxillae and palatines grow toward the midline. The secondary palate also creates a new passageway through the nose, the nasopharyngeal passage (Fig. 3), which lies above the secondary palate (its floor) and beneath the nasal capsule (its roof, where ossified). Orthonasal air passing through the ethmoid recess now courses along the nasopharyngeal passage before emptying into the back of the mouth via posteriorly displaced choana; retronasal air retraces this path in the opposite direction. In non-mammalian cynodonts the nasal capsule was entirely cartilaginous, but it became ossified in mammals where it took on a remarkable diversity of form as it adapted to contain the elaborate respiratory and olfactory turbinals.

In the oral cavity, the secondary palate forms a rigid roof against which the tongue can move food items toward the cheek teeth for processing or toward the esophagus for swallowing, and as a structural element increases the occlusal forces that the face can withstand (Crompton, 1989; Kemp, 2005). The secondary palate is fundamental to the new process of oral food processing or mastication. The oral and nasal chambers become confluent at the choana, which opens into the pharynx behind the tooth row, and orthonasal air enters the pharynx at the back of the mouth. Exhaled air can now take two routes, either through the mouth, as in panting, or back through the nasal chamber, via the retronasal pathway to re-cross the olfactory epithelium before exiting through the nares.

In Procynosuchus, the basal-most cynodont, the maxillae and palatines form a pair of shelves that extend two-thirds the length of the dentition (Fig. 7E). The shelves grow medially, but fail to meet on the midline and a narrow channel separates them from the vomer. This condition resembles the human congenital deformity known as cleft palate, in which the maxillae and palatines fail to grow sufficiently to meet on the midline, and a longitudinal slot affords a narrow conduit for some air to pass vertically between the nose and mouth. This was probably the case in Procynosuchus (Kemp, 1979). In all more derived cynodonts, the maxillae and palatines meet on the midline beneath the vomer to form a complete secondary palate (Fig. 7 F–I). By the origin of Mammaliaformes (Fig. 7I), the secondary palate extended to the back of the tooth row.

The Cynodont Dentition

The namesake feature of Cynodontia is their ‘dog-like’ dentition, in which a long canine separates simple incisors in front from complex molariform teeth that line the cheeks. The molariform teeth occluded in irregular facets, actively masticating food before it is swallowed. The basic plan of what would become the tritubercular molar was set, in which the crowns have three longitudinally aligned principal cusps, with the tallest in the middle, and an encircling ring of tiny cusps at their base known as cingula. Wear facets on the principal cusps are marked by micro-striations, providing evidence that upper and lower molariform teeth occluded while processing different types of food. Mastication shreds, dices, crushes, grinds, grates, chops, tears, rips, minces, pulverizes, and generally triturates food items before they are swallowed. This speeds the rate and degree of caloric return while liberating a new domain of information for analysis by the tongue and nose. In concert with the secondary palate, mastication made ortho-retronasal olfaction possible.

These developments initiated an episode of unprecedented dietary diversification reflected in a tremendous acceleration in rates of dental evolution. To frame this in a quantitative perspective, consider the basis of a series of phylogenetic analyses. Gauthier et al. (1988) scored 207 characters across 29 taxa that characterized skeletal variation in basal amniote clades (including non-mammalian synapsids), and only 8% (17) of these characters reflect dental variation. But for data matrices designed to capture variation among extinct cynodonts and basal mammals, Meng et al. (2006) scored 435 total osteological characters for 58 taxa, and 25% (108 characters) reflect dental variation. Building on that matrix, Ji et al. (2006) scored 445 characters for 103 taxa, and found 39% (173) of the characters as describing dental variation. In a matrix of 3660 osteological characters for 86 fossil and extant mammaliaforms, O’Leary et al. (2013) found 40% of total osteological phenomic variation (1450 characters) to reside in the dentition. Amplifying this diversity is the finding that homoplasy affects the cynodont dentition to a special degree. In a study of cynodont and basal mammal relationships, Rowe (1993) found in a matrix of 151 characters for 24 taxa that there was 30% more homoplasy in cynodont dental characters than in either the skull, the postcranium, or the combined skeleton.

There is hardly a branch on the cynodont tree that lacks its own unique molariform crown pattern. In most clades the dentition reveals the finest levels of taxonomic diversity and presents the diagnostic characters by which living and extinct species are identified. Evolutionary variability in the dentition parallels the high variability in the mammalian OR genome (Niamura, 2009, 2012) and is consistent with the view that olfactory ecology was a primary influence on the shape of mammalian diversification (Hayden, et al., 2010). In summary, retronasal smell gives added insight into the significance of dentition in relation to feeding behavior and evolution.

Associated changes in the mandible indicate a shift in the force vectors of the adductor musculature. Primitively the greatest bite forces were exerted at the front of the mouth for grasping prey. In cynodonts, reorganization of the mandible and zygomatic arch indicate that the mammalian temporalis and masseter were differentiated, and that the largest mandibular forces were shifting toward back of the tooth row for mastication (Gregory, 1953; Crompton, 1989; Kemp, 2005).

The Cynodont Cranio-Vertebral Joint

In basal synapsids, the skull articulates with the neck via a single spherical occipital condyle that is positioned beneath the foramen magnum. In cynodonts, the basioccipital largely recedes from the joint, and a ‘double occipital condyle’ is formed by the right and left exoccipitals which are positioned at the ventrolateral quadrants of the foramen magnum. The new articulation expanded the degree of stable dorsoventral excursion of the head on the neck without impairing passage of the spinal cord through the foramen magnum and along the cervical neural canal (Jenkins, 1971). It also suggests that the head was habitually held at a tilt with the nose toward the ground. Many mammals target their noses towards the ground and move their heads rapidly from side to side in scent-tracking and scent-guided navigation. More agile head movement potentially enabled cynodont olfaction to assert its importance in tracking, navigation and geographic memory.

Diaphragmatic Ventilation

In basal synapsids, the dorsal vertebrae are undifferentiated and bony ribs extend from the neck to the pelvis. In cynodonts, distinct thoracic and lumbar regions become differentiated in which long ribs persist on the anterior thoracic vertebrae, while the posterior three to five ribs form attenuated processes that fuse to their respective neural arches (Fig. 8). Differentiation of separate thoracic and lumbar regions marks the development of a muscular diaphragm, which separated the thoracic and abdominal cavities and initiated onset of the stereotyped vacuum-chamber or bellows-like tidal diaphragmatic ventilation of mammals

Figure 8.

Skeletons drawn to scale of Lycaenops (a Permian gorgonopsian), Thrinaxodon, and Morganucodon. Note the differentiation of thoracic and lumbar vertebrae in Thrinaxodon and Morganucodon skeletons, indicating presence of the diaphragm (modified from Dingus and Rowe, 1998).

While stationary or at rest, ventilation in living mammals is driven by the diaphragm (Bramble, 1989, Alexander, 2003). Chewing food is mostly a stationary action, thus ortho-retronasal olfaction is driven by diaphragmatic ventilation. The rapid sniffing so characteristic of many mammals in exploring their olfactory environments (Stoddart, 1980; Shepherd, 2012) is mostly done between steps or when moving slowly, and is also driven by the diaphragm. In basal cynodonts, the proximal ends of the thoracic ribs are flattened and imbricate in a condition unknown in living mammals. Hence their modes of breathing and locomotion were not entirely modern, but the important new capacity of diaphragmatic ventilation was introduced.

To summarize, in Late Permian cynodonts the structural basis of ortho-retronasal olfaction was established with the origin of the secondary palate and occlusal dentition, and with reorganization of the cranio-vertebral joint and the introduction of diaphragmatic ventilation. It was associated with a first modest pulse in encephalization that primarily affected the pyriform cortex. The principal question with the origin of cynodonts concerns the degree to which new cortical connections and associations were established between the multiple input modes that are integrated in mammalian ortho-retronasal olfaction. That all the skeletal equipment for ortho-retronasal olfaction was now in place suggests that the trend toward multi-layered neocortex and the multisensory integration that we refer to as flavor had begun its emergence in basal cynodonts. With EQs measuring only ~0.20, the first cynodonts fell considerably short of the capabilities inferred to have been present in the ancestral mammal (Rowe et al., 2011). Nevertheless, neurophysiological evidence reviewed below suggests that this was not a ‘simple’ cortex and was capable of higher orders of associations underlying discrimination, learning, and memory.

Triassic Cynodont Diversification

Over the next ~50 million years of Triassic cynodont history there is little evidence of further changes in relative brain size or structure. We doubt that cortical evolution was static, but it was insufficient to alter the braincase structure within the resolution of modern imaging techniques. Skeletal variation mostly involved successive modifications of the masticatory and locomotor systems. In the former system, the post-dentary bones became increasingly decoupled from mastication, reduced in size and dedicated to higher-frequency audition although retaining their attachment to the mandible.

Several other refinements of the masticatory apparatus pertain to ortho-retronasal olfaction, and these innovations preceded a second pulse in encephalization in the last common ancestor of Mammaliaformes (Node 8, below), in which the neocortex almost certainly emerged, and set the stage for the subsequent origin of crown Mammalia.

At Node 2 (Fig. 6 - unnamed), closure of the secondary palate was completed as the right and left maxillae and palatines sutured together along the midline. More powerful occlusal forces are indicated by successive increases in height of the coronoid process of the dentary, the extent of the masseteric fossa, thickening of the zygomatic arch, and expansion of the temporal fenestra. (Gregory, 1953; Kemp, 2005). Postcranial modifications suggest more agile and forceful locomotion, and greater ventilation capacity.

At Eucynodontia (Node 3), a key innovation appeared in the periodontal ligament. The teeth in basal cynodonts had short open roots that occupied shallow sockets and were held to the jaws by a ring of bone that surrounded the crown (Crompton, 1963; Rowe et al., 1995). In eucynodonts, the molariform teeth have long roots that close around the dental nerve during maturation and are implanted into deep sockets and held in place by a periodontal ligament (Rowe, 1993). Eucynodont fossils are commonly found in which the teeth slipped from their sockets before burial or, if the jaws still hold their teeth, inevitably there is a thin sheet of matrix around the roots, ancient sediment that filled the space once occupied by the ligament.

The periodontal ligament enables precise occlusal relationships to develop between upper and lower teeth during ontogeny (Noble, 1969; Ten Cate, 1969). In living mammals, tooth crowns erupt first, and opposing teeth twist and rotate and adjust to one another in forming consistent occlusal relationships. Implantation of the roots into their sockets follows eruption of the crown, and only after the occlusal relationship between crowns has formed do the roots lock the teeth in place with the periodontal ligament. It is this spatial plasticity as crowns erupt that dentists exploit in using braces to straighten and adjust the maturing teeth in adolescent humans (Wise and King, 2008). The ligament also serves as a shock absorber enabling more powerful occlusal forces. Its innervation supplements information from the crown on the texture of masticated food items via branches of the trigeminal nerve, which also convey somatosensory information from the oral cavity and tongue, and control most of the muscles involved in chewing and some of the muscles of swallowing (Barghusen, 1986).

Accompanying this innovation was a reduction in the rate and mode of tooth replacement. The primitive cynodont pattern of continuous alternating replacement of postcanine teeth throughout life occasioned frequent disruption of occlusion (Crompton, 1963). Eucynodonts adopted a pattern of consecutive replacement, and slowed the rate of replacement such that fewer tooth generations erupted from each socket and consistent patterns of occlusal facets between upper and lower molariform teeth could form (Rowe, 1993).

Node 4 (unnamed) is the taxon stemming from the last common ancestor mammals share with the Early Triassic Diademodon. Pre-cladistic accounts portrayed cynodont history as two grand radiations, one of herbivorous ‘gomphodonts’ with broad tooth crowns, and the other of persistently predatory cynodonts from which mammals ultimately descended (e.g., Hopson, 1969; Crompton, 1972). The first phylogenetic analyses quickly established that some ‘gomphodonts’ such as Exaeretodon and tritylodonts (below) are more closely related to mammals than to Diademodon and that the ‘gomphodonts’ are a paraphyletic assemblage of presumed herbivores (e.g., Rowe, 1988, 1993; Gauthier et al, 1988). In this light, it appears that dental occlusion and mastication enabled a far greater measure of dietary diversification than previously believed, and that the adoption of herbivory occurred multiple times among cynodonts.

At Node 4 the cheek teeth now have two roots, each surrounding its own dental nerve and root canal. In early forms like Diademodon, a thin web of bone still connected the roots, but this was soon lost and the roots became fully divided. The crowns in these cynodonts were generally expanded and closely packed, forming in some taxa a broad, continuous occlusal surface and one that provided greater levels of textural information about food items (Rowe, 1993).

At Node 5 escalation of the role of the trigeminus in feeding behaviors is evident in the presence of a partial bony floor beneath the cavum epipterycum. This space contains the trigeminal ganglion, and its partial enclosure marks expansion of the ganglion (Bonaparte, 1966; Rowe, 1993). In crown mammals the cavum epipterycum becomes completely enclosed by the bony braincase in early development, although it remains external to the meninges and outside of the cavum cranii proper.

Mammaliamorpha (Node 6)

This clade stems from the last common ancestor mammals share with the Late Triassic tritylodontids (Rowe, 1988), and its origin coincides with a number of important transformations that preceded a second pulse in encephalization (Node 7, below). The first mammaliamorphs became miniaturized (Fig. 8), and as adults occupied the one or two smallest orders of vertebrate size magnitude. A few descendant lineages regained much larger size during the Mesozoic, but for the next ~135 million years, nearly the entire history of mammaliamorphs including crown mammals played out in tiny animals (Kielan-Jaworowska, et al., 2004). Only in the Cenozoic did numerous mammalian clades independently evolve much larger body sizes, but even today the greatest diversity is in small species. With miniaturization, early mammaliamorphs encountered greater spatial and environmental heterogeneity. Entry into new microhabitats corresponds to diversification in diet, activity patterns, and life history strategies (Harvey et al., 1980; Mace et al., 1981). New food items such as seeds, grains, fungi, small fruiting bodies, and small invertebrates were available for the first time. Ortho-retronasal olfaction now included high resolution somatosensory information provided by the oral field of the trigeminus, enabling early mammaliamorphs to explore and exploit more thoroughly their new microhabitats.

In the skull, the medial wall of the orbit was closed as the orbitosphenoid became co-ossified with the sheets of thin bone contributed by the palatine and frontal (Sues, 1986), and the rear parts of the nasal capsule were ossified for the first time (Kielan-Jaworowska, et al., 2004). Together with elaborate modifications of the side wall of the braincase (Rowe, 1988, 1993), this likely reflects its own modest pulse in encephalization. However, efforts to image endocasts in tritylodontids have been unsuccessful. Additionally, negative allometry of the auditory ossicles indicates that they were increasingly decoupled from their role in mastication in favor of high-frequency audition (Rowe, 1996a, b). The cochlea had begun to elongate (Kielan-Jaworowska, et al., 2004), but its full coiling occurred much later, within various mammalian subclades (Luo et al., 2011).

Miniaturization was accompanied by remodeling of the postcranial skeleton, in which the joint surfaces were more precisely sculpted, and the trochanteric attachments for limb muscles resembled those found in mammals. Locomotion at small size is metabolically far less expensive than at larger size, and climbing vertically costs little more than locomotion over flat surfaces (McMahon and Bonner, 1983). There is also a regular change in the mechanical advantage muscles have about the joints of the skeleton as a consequence of the stability of the joints under loads determined by inertia and gravity, and flexion angles at joints increases as size decreases. This implies that muscle spindles and joint proprioceptors were recording more information than before, as a new level of agility emerged. Herbivores comprise the largest portion of biomass within modern mammalian communities (Eisenberg, 1990), and even a partial shift to the new trophic level of primary consumer may have supported larger populations of early mammaliamorphs than in earlier cynodonts.

Mammaliaformes (Node 7)

A second pulse of encephalization occurred in Mammaliaformes (Rowe, et al., 2011), which is the clade stemming from the last common ancestor shared by Morganucodon and Mammalia (Rowe, 1988). The endocranial cavity is fully enclosed, and endocasts indicate the brain of Morganucodon (Fig. 9A–D) to be nearly 50% larger than more primitive cynodonts, with an EQ of ~0.32. The olfactory bulb and cortex are by far the regions of greatest expansion. A deep annular fissure separates the inflated olfactory bulb from the inflated olfactory cortex, which by this time was much wider than the hindbrain. The cortex and cerebellum cover the midbrain and the pineal stalk. The cerebellum is also enlarged and suggests expansion of the basal nuclei, thalamus and medulla, and the spinal cord is also thicker. The mammaliaform brain now resembles the shape of a mammalian brain more than the brain in basal cynodonts.

Figure 9.

Digital endocasts of Morganucodon (A–D) and Hadrocodium (E–H) in dorsal (A, E), ventral (B, F), right lateral (C, G) and left lateral (D, H) views. Abbreviations: Cb, Cerebellum; Fr1, Fr2, postmortem fractures displacing parts of endocast; Fan, annular fissure; Hyp, hypophysis; Iam, internal acoustic meatus; II, optic nerve; Ncx, Neocortex; Ob, olfactory bulb; Pcx, olfactory (pyriform) cortex; Pfl, paraflocculus; Sss, superior sagittal sinus; V, trigeminal nerve (from Rowe et al., 2011).

Integumentary evidence from the remarkably preserved Middle Jurassic mammaliaform, Catorocauda lutrasimilis (Ji et al., 2006) indicates the presence of a pelt of modern aspect, with both guard hairs and vellus underfur. Body hair develops from migrating neural crest cells that condense into tiny placodes that mature into hair follicles equipped with at least three types of mechanoreceptors. The genetic basis of hair is poorly understood, but its impact on neocortical maturation is clear. Tactile signals induce the formation of sensory and motor maps on the primary somatosensory field, implying that the neocortex was differentiated in basal mammaliaforms (Rowe, et al., 2011). Increased olfactory sensitivity, and improved tactile resolution and motor coordination, account for much of this second pulse in relative brain size. The presence of a pelt suggests that early mammaliaforms were also endothermic, and the ontogeny of endothermy in living mammals implies parental care (Rowe et al., 2011). Endothermy may have been a consequence of both an increased surface-to-volume ratio at miniature size and greater encephalization. A large brain is metabolically expensive to maintain (Allman, 1990), but because metabolism is largely under hormonal control it did not itself directly drive encephalization. Despite their size, by the end of the Early Jurassic, small mammaliaforms had a global distribution (Kielan-Jaworowska et al., 2004).

At Node 8 (unnamed), the Early Jurassic fossil Hadrocodium (Luo et al., 2001) indicates a third discrete pulse in encephalization, raising its EQ to ~0.5, a level that lies within the range of crown mammals (Rowe et al., 2011). Most of this increase in relative size is in the olfactory bulbs and pyriform cortex (Fig. 9E–H). An extraordinary morphogenic consequence of the expanded olfactory cortex is that the auditory chain was disrupted during ontogeny, and those ossicles directly involved in hearing were detached from the mandible and suspended exclusively from beneath the braincase.

Discovery of the Early Cretaceous basal mammal Yanoconodon allini (Luo, 2007; Luo et al., 2007) appeared to belie the hypothesized ontogenetic relationship between cortical expansion and detachment of the middle ear ossicles (Rowe, 1996a, b), because it has both a large brain and an ossicular chain connected to the jaw. However, numerous features of the only known skeleton attest to its immaturity at time of death, and that it corresponds to a 3 to 4 week old opossum in which detachment of the ossicles has yet to occur. Lack of fusions in the atlas, axis, and along the rest of the vertebral column, and between pelvic elements indicate that Yanoconodon presents an ontogenetic transitional stage, rather than a phylogenetic intermediate, and that this individual died before the position of the ear ossicles matured. The larger point is that cortical expansion had a remarkable phylogenetic impact on cranial architecture in mammaliaformes and crown mammals, and that it explains one of the historically most problematic transformations in early mammalian evolution.

In Hadrocodium, the cerebellum has also expanded to such a degree that the occipital plate bulges backwards, enclosing a relatively large foramen magnum and spinal cord. Only a few features separate Hadrocodium from crown mammals. Most importantly, it lacks ossified turbinals, and large pterygoid processes indicate that its bilateral chewing and swallowing mechanics were transitional to those in crown Mammalia. Once the brain reached this level of relative size, it continued to diversify as it supported diverse sensory partnerships that evolved in the different mammal clades. That information was an essential commodity to the diversification of mammals is evident in the fact that further increases in encephalization occurred independently in nearly all of the major clades (Jerison, 1973; Rowe et al., 2011).

It is noteworthy that evolutionary decreases in encephalization are rare. One of the best documented cases is in the platypus lineage, where more highly encephalized Cenozoic fossils indicate reduction of both the olfactory bulb and overall EQ in the evolution of the living platypus Ornithorhynchus (Macrini et al., 2006). Decreases in encephalization are also associated with domestication in various mammalian species (Kruska, 2007).

The Origin of Mammalia (Node 9)

An ossified derivative of the embryonic nasal capsule, known in total as the ethmoid bone and including the elaborate skeleton of the ethmoid turbinals, arose with origin of Mammalia and followed the developmental pathway described above (Rowe, et al., 2005). Comparisons among mammalian subgenomes suggest ~ 1200 OR genes were present in mammals ancestrally (Niimura, 2009, 2012) and their full expression was facilitated by the new surface area provided by the turbinals.

It seems remarkable that an order of magnitude expansion in OE surface area could occur without a corresponding increase in the size of the nose. This tempts speculation that without compensation in the visual system, expansion of the nose dorsally and laterally would disrupt the forward binocular visual field; while caudal expansion is limited by the optic chiasm, against which the ethmoid abuts. The secondary palate and dental occlusion may have also constrained organization of the facial skeleton, reflecting the critical role that mastication had long since assumed in cynodont physiology. Speculation aside, the developmentally adaptive nature of trubinal growth enables it to support both patent airways and a ten-fold expansion of OE within a confined space. It also maintains spatial identity of the expression loci for specific OR genes as the ORs themselves are renewed throughout ontogeny, and it helps funnel new OR axons toward their target glomeruli. That all this occurs without radically altering a plan of facial architecture established in the first cynodonts underscores the integration of mastication as part of the larger system of ortho-retronasal olfaction.

Three Layer Association Cortex and the Origins of Neocortex

We have seen that the primitive skeletal structure of early amniotes was profoundly transformed in the emergence of early mammals, and that ortho-retronasal olfaction played a central role in linking correlated transformations in brain size, the masticatory system, and in the postcranial skeleton. The neocortex arose in this environment of vastly increased airborne and internal olfactory information, a major escalation in somatosensory information, and additional sources of new information from the ear. Finally, and crucially, these small creatures were endothermic, which supported a constantly active cortically-controlled motor system that mediated prolonged foraging behavior to support their high metabolism, their young, and survival among larger predators.

The relative size of the neocortex in early mammals was small. Figure 10 shows the structure of the brain inferred to have been present in the ancestral mammal, in which pyriform cortex was much larger than the small neocortical areas subserving other sensory and motor systems (Molnár et al, 2014). The somatosensory area subserving the sensory function of hair was introduced prior to the origin of mammals (in basal Mammaliaformes), whereas successive increases in pyriform cortex can be traced from the origin of cynodonts to the diversification of mammals. Given the evidence for an evolutionary increase in the size of pyriform cortex, we next consider to what extent principles of neocortical organization may have been developed from those involved in olfactory cortical function.

Figure 10.

Depiction of the mammalian ‘ancestral forebrain cortex’ (from Molnár et al., 2014). Abbreviations: OF, orbitofrontal area; MF, medial frontal area; S1, S2, RS, CS: primary, secondary, dorsal and caudal somatosensory areas; g, gustatory area; V1, V2, t: primary, secondary and temporal visual areas; Aud, auditory areas; CCv, CCd: ventral and dorsal cingulate areas; RSg, RSa, retrosplenial granular area and agranular areas; SC, superior colliculus; IC, inferior colliculus.

Olfactory or pyriform cortex appears to have had a similar neural organization in turtles and lizards (Ulinski, 1983; Bruce, 2007, 2009; Bruce and Braford, 2009) as in monotremes, marsupials and placentals (Ashwell, 2010, 2013; Shepherd, 2011), supporting the inference that this organization was present in amniotes ancestrally. In brief, studies of opossum and rodent have shown that in olfactory cortex, the output fibers from the olfactory bulb course over the surface of the cortex, emitting collaterals that terminate on the most distal dendrites of the pyramidal cells (Fig. 11). Their action is excitatory onto spines of these cells, as well as onto smooth dendrites of interneurons which feed inhibition forward onto the pyramidal neurons. The activated pyramidal cells through their axon collaterals feed back recurrent excitation onto themselves and neighboring pyramidal neurons. They also excite interneurons which feed back inhibition onto themselves and neighboring pyramidal neurons. Thus in the two critical operations of the circuit – processing the input and the output – excitation and inhibition are balanced. This circuit is shown in Figure 11.

Figure 11.

Top: Olfactory cortical areas on the ventrolateral surface of the cerebrum of the rat. Abbreviations: AOC, anterior olfactory cortex; ctx, cortex; olfac tub, olfactory tubercle. From Neville and Haberly (2004). Bottom: Microcircuit organization of the mammalian piriform (olfactory) cortex. Abbreviations: LOT, lateral olfactory tract; SP, superficial pyramidal cell; DP, deep pyramidal cell; S, stellate cell; C, centrifugal fiber. Arrows indicate the direction of flow of activity. Open profiles: excitatory synaptic action; filled profiles: inhibitory synaptic action. After Haberly and Shepherd (1973).

It is tempting to assume that olfactory cortex is equivalent to primary sensory cortex in other senses. However, Haberly (1985) showed many years ago that the intrinsic organization of olfactory cortex, with its long association fibers, was much more similar to higher association cortical areas, for example the face area of inferotemporal cortex. This is now widely accepted. As noted above, olfactory cortex takes input from the olfactory bulb in the form of an odor image and transforms it into a central representation as an odor object (Shepherd, 1991; Wilson and Stevenson, 2006). The evidence from mammals, lizards, and turtles indicates that this higher order function was present at the origin of Amniota, and was elaborated in basal cynodonts in which occurred the first pulse in encephalization associated with evolution of the uniquely mammalian neocortex.

A second major cortical area is the hippocampus, differentiating from the medial wall in amniote forebrains. Anatomical and physiological studies have shown that across amniotes the neurons are similar to those in the olfactory cortex, with similar interconnections for excitation and inhibition (Connors and Kriegstein, 1986; Haberly, 2001). Of special note is the tendency to burst firing, believed to be analogous to the dorsal hippocampus in rodents with susceptibility to seizure activity. As in rodent hippocampus, tetanic stimulation of the septum in the turtle Pseudemys scripta gives rise to heterosynaptic long-term potentiation (LTP) (Muñoz et al., 1998).

Numerous studies show a close similarity between the intrinsic organization of the hippocampus and the basic organization of the olfactory cortex, in terms of layering of inputs on the apical dendrites and long association fibers (Fig. 12; Neville and Haberly, 2004). Since the inputs to the hippocampus consist exclusively of central sites in the limbic regions, it is clear that three-layer hippocampus from the start of amniote and mammalian evolution was devoted to higher order processing such as learning and memory. This adds to the evidence that three layer cortex is not a primitive ‘simple’ cortex, but rather operates at the level of higher order associations underlying discrimination, learning and memory. Note that the connections of olfactory cortical and hippocampal pyramidal cells in turtles, lizards and mammals are almost exclusively intratelencephalic, that is, they are restricted to the telencephalon (cortex and basal ganglia) and do not project to lower levels of the brain stem and spinal cord, a point that will be important in comparing them with neocortex (see below).

Fig. 12.

Similarities between the laminar organization and horizontal connections of olfactory cortex and hippocampus. Abbreviations: 1. OB, olfactory bulb; AOC, anterior olfactory cortex; APC_V, ventral anterior piriform cortex; APC_D, dorsal anterior piriform cortex; PPC, posterior piriform cortex; sup, superficial. 2. EC, entorhinal cortex; DG, dentate gyrus; prox, proximal; dist, distal; SL-M, stratum lacunosum- moleculare; SR, stratum radiatum; sup, superior; SP, stratum pyramidale; SO, stratum oriens. From Neville and Haberly (2004).

This leaves dorsal cortex as the anlagen, the precursor, of neocortex in mammals. A pioneering anatomical study by Smith et al. (1980) of the turtle Pseudemys scripta reported connections through an interneuron that provide for feedforward and lateral inhibition. This was followed by an electrophysiological study which incorporated feedback and lateral excitation and inhibition into the local circuit (Kriegstein and Connors, 1986; Connors and Kriegstein, 1986) (Fig. 13). The close similarity across amniotes of this local circuit to the olfactory and hippocampal circuits pointed to a ‘basic circuit’, some would call it a ‘canonical circuit’, common to all three forebrain regions (Kriegstein and Connors, 1986). This does not mean that each region does not have its own fine tuning for its particular types of input, but rather that there is a basic framework common to all three. Since dorsal cortex in turtles (Ulinksi, 1983) and lizards (Bruce, 2007, 2009) receives input from the visual pathway, there is a tendency to regard it as a primary visual cortical area, equivalent to mammalian V1. However, the comparison with three-layer hippocampal and olfactory cortices suggests that, like those regions, dorsal cortex performs higher level association on the visual input. In this view, three-layer dorsal cortex which gave rise to neocortex was not a ‘simple’ cortex for low-level processing, but rather had an organization that subserved high-level association functions analogous to those in pyriform cortex and hippocampus.

Fig. 13.

Left, above: Dorsal view of the brain of the turtle Pseudemys scripta. Abbreviations: DC, dorsal cortex; H, hippocampus; P, pyriform (olfactory) cortex. Below, cross-section of the forebrain at the level of the arrow in top diagram, showing the relative positions of pyriform (P), hippocampal (H), and dorsal cortex (DC). From Connors and Kriegstein (1986). Right: Summary of the microcircuit organization of turtle dorsal cortex. Thalamocortical afferent volleys (1) excite pyramidal cell dendrites (a) and also inhibitory stellate cells (b). Stellate cell-pyramidal cell contact (2) mediates feedforward inhibition. Pyramidal cell output mediates reciprocal excitation between pyramidal cells (3) as well as feedback inhibition (4). Stellate-stellate cell contacts mediate inhibition (5). The pyramidal cell axons provide output (6) After Kriegstein and Connors (1986). Open profiles: excitatory synaptic action; filled profiles: inhibitory synaptic action.

This conclusion from comparative physiology is supported by the comparative anatomy of living amniotes, which suggests that basal amniotes possessed a forebrain with three clearly demarcated regions, viz. a lateral olfactory cortex; a medial hippocampus; and the dorsal intermediate part that undergoes voluminous expansion over the course of mammalian history. This comparison supports the physiological evidence that the dorsal intermediate part is the region from which neocortex was successively elaborated in the earliest cynodonts and mammaliaforms from a three-layer cortex of basal amniotes and early synapsids.

These considerations suggest that visual cortex in the ancestral amniote may have functioned as a higher order visual association area rather than a primary visual area. This adds to the interest of the pyriform cortex as the best-studied example of the type of higher order processing carried out by so-called ‘simple’ three-layer cortex. As suggested by Fournier et al (2014: p.122), “It could be that DCx [dorsal cortical] neurons are selective to high-order correlations, and process spatio-temporal sequences of distributed visual cues in a manner similar to how PCx [pyriform cortex] processes spatio-temporal activation of specific glomeruli”.

Pyriform cortex thus appears to have evolved to process higher-order associations within the high-dimensional space representing odor molecules, pushed ever higher by the expansion of the OR subgenome in mammals and the new volatiles released by retronasal smell. A parallel higher-order organization in dorsal cortex appears to have been amplified in neocortex. The comparative and paleontological evidence thus suggest that higher order association functions were present at the inception of mammalian neocortex. The olfactory cortex expanded for the higher-level combinatorial associations associated with the expansion of the OR subgenome and retronasal volatiles. The hippocampal cortex expanded in processing limbic inputs involving more complex learning and memory. And the neocortex differentiated new cell types and layers not only building more sophisticated circuits for processing ortho-retronasal olfaction, vision, somatosensation and audition, but also for novel neocortical control of lower level motor connectivity needed for the survival of small endothermic, continuously active, omnivorous mammals. This is consistent with the transfer of much of visual processing from the collicular level in reptiles (and presumably basal amniotes) to the cortical level in mammals. Finally, the explosion of higher order cortical areas, numbering 50 or more in some mammals, was associated with a corresponding explosion in connectivity between cell types interacting within the telencephalon as well as connecting to downstream sites of input and output (Harris and GMG Shepherd, 2015).

How did these properties provide the basis for the emergence of neocortical neurons and circuits? It will be useful to return to the early pulses of encephalization in cynodonts and compare the likely changes in neuronal organization that occurred in developing from three layer dorsal cortex to neocortex in these creatures. Figure 14 provides simplified diagrams, showing on the left a single type of pyramidal cell (PC) of three layer dorsal cortex, and on the right the three pyramidal cell types that current studies identify as characteristic of neocortex: intratelencephalic (IT) cells (whose projections remain within the cortex and basal ganglia); pyramidal tract (PT) cells (including all their connections throughout the brain stem and spinal cord); and corticothalamic (CT) cells (see Reiner et al., 2010; Fame et al., 2011; Harris and Shepherd, 2015). The pyramidal cells (PC) of dorsal cortex also give rise to association fibers, that is, they function as IT cells (IT in the diagram). They give rise to motor output through their connections to the basal ganglia, whose cells project further to the brain stem tectum (Reiner et al., 1998) with further connections to the spinal cord. Reptiles thus lack the direct control over the spinal cord motor neurons that is made possible in mammals through the PT cells.

Fig. 14.

Comparison of the main types of pyramidal cells in three-layer cortex and the neocortex. Left: Three layer cortex: PC, pyramidal cell; INT, interneuron; VIS, visual input; IT, intratelencephalic (connections to cortex (C) and basal ganglia (BG)). Right: Neocortex: Intratelencephalic (IT) pyramidal cells (shown in red) are analogous to association pyramidal cells in 3 layer cortex (connections to cortex (C) and neostriatum in the basal ganglia (BG)). They may be found throughout layers 2–6 (dotted line). New types of pyramidal cells (shown in blue) in neocortex are stellate (ST) cells (layer 4) receiving input from thalamocortical (TC) fibers, pyramidal tract (PT) cells in layer 5B, and corticothalamic (CT) cells in layer 6. IT cells are by far the most numerous pyramidal cell type in neocortex. Not shown are intrinsic circuits for feedback and lateral excitation and inhibition, also adapted from 3-layer cortex, as described in relation to Figs 8–11. Diagram adapted in part from Molnár et al., (2006), Reiner et al., (1998, 2010), Fame et al., (2011), and Harris and Shepherd (2015).

On the basis of the analysis of the pyramidal cell (PC) in three layer cortex, we hypothesize that its association functions appear to provide a template for the association functions of the intratelencephalic (IT) pyramidal cells distributed throughout layers 2–6 of the neocortex (indicated by the red coloring of both types in the diagram). From this perspective, the critical changes within neocortex were that the IT pyramidal cells and their association connections became greatly amplified within and between the layers, more specialized in their outputs (corticocortical connections for superficial IT cells and neostriatal outputs from deep IT cells), and extended to the multiple cortical areas in both hemispheres. On the output side, what is new are outputs through the pyramidal tract (PT) cells of layer 5B to many subcortical areas, and stronger connections to thalamus through corticothalamic (CT) cells of layer 6. On the input side, the pathway involving thalamocortical fibers and stellate (ST) cells is special for neocortex.

How does this hypothesis relate to current studies of the neurogenesis of cortical neuron types (Molnár et al., 2014; Shibata et al., 2015)? In all vertebrates, epithelium in the ventricular zone (VZ) is the primary embryonic neurogenic source, and in mammals the major source of cells in the deeper neocortical layers (5–6). Mammals in addition have an adjacent neurogenerative layer known as the subventricular zone (SVZ), controlled by different genes (Cheung et al., 2010; Franco et al., 2012). The SVZ is a source of gliogenesis, and of neurons in the upper neocortical layers (2–4). However, other studies indicate that the same population of progenitor cells can sequentially generate the neuron types in all layers (Guo et al., 2013). The contribution of the SVZ in neuronal production seems to grow in evolution as the complexity of the neocortex increases (Cheung et al., 2010; Molnár et al., 2014).

Correlation of these studies with the specific neuronal types shown by physiology is needed. The studies cited suggest that upper layer cells include IT cells, whereas deeper layer cells include the PT and CT types. Further studies are needed to determine how the SVZ in mammals generates the IT cells and their association circuits that appear to be based on the three layer PC template. We postulate that the SVZ was present in cynodonts, the critical step toward generating the multiple cell types and laminae of the earliest neocortex (Fig. 14). It appears most likely to have emerged in basal mammaliaforms, especially in Hadrocodium, owing to the relative increase in the size of its brain.

More evidence is needed on whether all neocortical elements were fully present in the earliest cynodonts. Early cynodonts display all the skeletal equipment for ortho-retronasal olfaction, inviting speculation on which if any of these neocortical elements developed or were present in some incipient form, and which were added secondarily in basal mammaliaforms with the next jump in encephalization. The amount of neocortex in the ancestral forebrain was likely quite limited (Fig. 10). One needs to understand more deeply the ecological niche these early mammaliaforms occupied and their sensory adaptations. The paleontological evidence indicates that the latter likely included heightened olfaction for widened olfactory signals on land and new volatiles through retronasal smell. The acquisition of fur appears associated with endothermy and represents a connectional invasion of new peripheral information. These changes facilitated continuous sensory and motor activity, graded at all times with the possible presence of prey or predator and the consumption of food. A critical change was from behavioral control through the brainstem, as in visual and auditory reflexes through the superior and inferior colliculi, respectively, to control at the forebrain level through thalamus and cortex. Association connections emerged for many new functions involved in social bonding, care of the young, and eventually higher cognitive functions. These new connections made the neocortex dominant in sensorimotor coordination and in the readout of higher level association cortex. This hypothesis based on paleontology and physiology thus needs to join the mainstream of anatomical, developmental, and molecular studies to focus on the earliest steps in the critical transition from 3 layer cortex to neocortex.

Ortho-Retronasal Olfaction and Human Neocortical Evolution

We turn finally to neocortical evolution in pre-humans and humans. Although the visual system has traditionally been regarded as playing a large role, and olfaction a minor role, recent evidence suggests that ortho-retronasal olfaction was critical for this phase too.

As we have seen, diversification of the masticatory system was linked to the system of ortho-retronasal olfaction, and much of the history of cynodont (including mammalian) diversification reflects elaboration of these two coupled systems. As we have argued, it is more apt to think of them as one integrated system, although the olfactory component is generally overlooked when studying mammalian masticatory evolution, and vice versa.

In haplorrhine primates, the clade to which humans belong, there was a loss of functional OR genes to about ~350 – 400 that affects the entire clade (Niiamura 2009, 2012). Entrenched in the literature is the presumption that this reduction corresponds to poor olfactory performance. However, a succession of new discoveries is altering this view. Keen olfactory perception was demonstrated in primates (Laska, et al., 2000) that challenges a simple relationship between numbers of olfactory receptor genes and olfactory performance (Gilad et al., 2005), and the precise role of increased primate encephalization is as yet unexplored in this regard. Next, discovery in the human olfactory bulb of a convergence ratio of 1 OR type to 16 glomeruli (Maresh, et al., 2008) indicates an eight-fold increase in glomeruli per OR, which may underlie an unprecedented olfactory enhancement. It was also recently shown that humans are capable of scent-tracking, much like the performance in other macrosmatic mammals, and that with practice their scent-tracking performance improves (Porter et al., 2007). These investigators suggest that the poor reputation of human olfaction may reflect behavioral demands rather than ultimate abilities. Most recently, it has been demonstrated that the lower limit on human odorant stimuli is one trillion, which is several orders of magnitude larger than the numbers of colors and tones that humans discriminate (Bushdid et al., 2014). Whether this last estimate applies beyond humans remains unknown.

Anatomical changes are also involved. In non-haplorhine primates the lamina transversalis forms the caudal floor beneath the sphenethmoidal recess of the nose, which is a blind space containing olfactory epithelium (Smith and Rossie, 2006). In haplorhines, this thin plate of bone fails to develop and the sphenethmoidal recess is open broadly to the nasopharyngeal passageway, affording greater exposure of olfactory epithelium to both orthonasal and retronasal air.

Airflow experiments on human cadavers demonstrated that orthonasal and retronasal air moves through the nose in different ways (Proetz, 1953). Since the nostril is of much narrower diameter than the choana, orthonasal air tends to flow in a laminar path across the nasal cavity during diaphragmatic breathing, while retronasal air takes on turbulent flow that carries more volatiles to all parts of the olfactory epithelium. Turbulence is caused as air passes through the large aperture of the choana and backs up behind the smaller nostril. With the loss of the transverse lamina, both orthonasal and retronasal air currents reach more of the olfactory epithelium, with the latter carrying the additional information liberated by mastication. Current experiments are underway to reconstruct the human oro- and nasopharynx in order to carry out quantitative modeling of airflows underlying human retronasal smell (Ni R, Michalski M, Zinter J, Ouellette NT, Brown E, Shepherd GM, unpublished observations).

If ortho-retronasal smell was strong in ancestral humans, how did this affect the evolution of the human brain? A key factor was the invention of controlled fire. In ‘Catching Fire: How Cooking Made us Human’, Richard Wrangham (2009) provides strong evidence that the most important property bestowed by domestic control of fire was the increase in energy obtained from cooked food, perhaps doubling or tripling both speed of ingestion and caloric return. This was critical for the brain, because pound for pound it is the most energy-demanding organ. Humans also used fire to modify their environments and inhabit higher latitudes and elevations, and it was central to the emergence of settlement and land use systems and the evolution of social systems and languages (Rolland, 2004). Hominids inherited the more general omnivorous primate diet, and adding meat-procurement and meat-eating moved them up the trophic chain to compete with other carnivores, adding yet another measure of complexity to their lives. Archeological associations of humans and fire date to more than one million years before the present. However, evidence that hominids controlled and domesticated fire and cooked food is not securely established until about 400,000 years ago, not long after the estimated divergence of Homo sapiens from Neanderthal, at ~500,000 years ago (Green, et al., 2006). Microfossils in tooth calculus demonstrate the consumption of cooked plants in Neanderthal diets (Henry et al., 2011). They had also adapted sophisticated lithic technology for hunting at least small animals (Rendu, 2010). Homo sapiens developed these skills in unprecedented degree.

From our perspective, ortho-retronasal olfaction was key to this role of cooked food. Cooking produces many new flavors that make the food more attractive. In ‘Neurogastronomy: How the Brain Creates Flavor, and Why it Matters’ (Shepherd, 2012) it is argued that flavor resides not in the food, but is created by brain circuits. Humans have more brain circuits for this task, and they inherited all the mechanical equipment necessary for ortho-retronasal olfaction (Fig. 15). The closeness of the oropharynx to the retronasal pathway in fact appears to make humans better adapted for retronasal olfaction than other mammals (cf. Fig. 1). The human retronasal pathway excelled in making available new domains of flavor from the cooked food, along with a multiplicity of brain circuits to create perceptions of food that humans want and crave.

Figure 15.

The dual human olfactory system. (A) Brain systems involved in smell perception during orthonasal olfaction (sniffing in). (B), Brain systems involved in smell perception during retronasal olfaction (breathing out), with food in the oral cavity. Air flows are indicated by dashed and dotted lines; dotted lines indicate air carrying odor molecules. ACC, accumbens; AM, amygdala; AVI, anterior ventral insular cortex; DI, dorsal insular cortex; LH, lateral hypothalamus; LOFC, lateral orbitofrontal cortex; MOFC, medial orbitofrontal cortex; NST, nucleus of the solitary tract; OB, olfactory bulb; OC, olfactory cortex; OE, olfactory epithelium; PPC, posterior parietal cortex; Pont, pontine taste nucleus; SOM, somatosensory cortex; VII, IX, X, cranial nerves; VC: primary visual cortex; VPM, ventral posteromedial thalamic nucleus. Not shown are numerous additional systems involved in motivation, memory, emotion, reward and language (from Shepherd, 2006).

In summary, ortho-retronasal olfaction was central to the early evolution of the mammalian brain, the expansion from three layer cortex to six layer neocortex, and finally to the human craving for flavorful high-energy food that led to the evolution of a much larger brain and a richly diversified, highly interconnected human neocortex. Smell thus was not relegated to being a minor sense in human evolution, but rather was at the core of how our brains evolved to make us human.