Neural Mechanisms Underlying Sex-specific Behaviors in Vertebrates

From invertebrates to humans, males and females of a given species display identifiable differences in behaviors, mostly but not exclusively pertaining to sexual and social behaviors. Within a species, individuals preferentially exhibit the set of behaviors that is typical of their sex. These behaviors include a wide range of coordinated and genetically pre-programmed social and sexual displays that ensures successful reproductive strategies and the survival of the species. What are the mechanisms underlying sex-specific brain function? Although sexually dimorphic behaviors represent the most extreme examples of behavioral variability within a species, the basic principles underlying the sex specificity of brain activity is largely unknown. Moreover, with few exceptions, the quest for fundamental differences in male and female brain structures and circuits that would parallel that of sexual behaviors and peripheral organs has so far uncovered modest quantitative rather than the expected clear qualitative differences. As will be detailed in this review, recent advances have directly challenged the established notion of the unique role of steroid hormones in organizing and activating male and female-specific brain circuits, and have uncovered new mechanisms underlying the neural control of sex-specific behaviors.

Historical perspective

The mating behavior of the three-spined stickleback represents one of the best described examples of chain reactions of sex-specific interactions [1]. The male stickleback initiates a zigzag courtship dance when detecting the presence of a pregnant female. The female reacts to the red color of the male by swimming aside, inducing the male to lead the female to its nest. The presence of the female in the nest releases a trembling behavior by the male, which leads to female spawning. Only after the eggs are spawned will the male release its sperm and fertilize the eggs. In other species such as frogs, crocodiles and songbirds, males produce an intense male-specific courtship song that entices the female to enter their territory and pick them as mate, while in rodents, intense olfactory investigation leads to male and female-specific sexual behaviors. Other social sex-specific behaviors in rodents include male-male and lactating female aggressive behaviors, and parental behavior in which females of most species invest significantly more time in taking care of the offspring. Thus, each species has evolved discrete communication strategies and behavior responses enabling individuals of each sex to identify each other, to successfully breed and to care for their progeny. The study of developmental and neural mechanisms controlling sexually dimorphic brain function has been the topic of a vast amount of research, leading to the identification of gonadal hormones as a major player in the establishment of sex-specific brain function. Moreover, these earlier studies uncovered discrete neuronal circuits and nuclei underlying sex-specific responses, and pointed to the central role of the vomeronasal pathway in governing the sensory control of sexual behavior.

The role of sex hormones

Arnold Adolph Berthold first investigated the role played by gonadal hormones in establishing sex-specific behavior. Berthold hypothesized that intact testes are necessary for the development of male-typical characteristics, leading him to conduct a series of experiments using castration and testes replacement in roosters. He found that males castrated as juveniles later showed deficits in adult behaviors such as aggression, mating and crowing. These castrated animals also failed to develop the large size and the plumage characteristics of normal males. Furthermore, Berthold discovered that the effects of castration could be reversed if the subject’s testes, or the testes of another male, were implanted into the body cavity of a castrated juvenile, giving the first indication that hormones secreted by the gonads play a critical role in the organization of sexually dimorphic behaviors [2].

A series of experiments on guinea pigs performed by William Caldwell Young confirmed the critical role played by gonadal hormones in establishing the basic pattern of reproductive behavior [2–4]. Young demonstrated that perinatal exposure of female guinea pigs to elevated androgens permanently suppressed their capacity to display feminine sexual behavior (defeminization) and significantly enhanced their display of masculine sexual behavior (masculinization). From these behavioral results it was suggested that the exposure to prenatal androgens had permanently altered the brain structures underlying sexual behavior and that, similarly to the development of peripheral sexual organs, androgens ‘organized’ the developing nervous system at a critical period of early development [5].

These findings led to the proposal that gonadal hormones have a dual role in the organization and activation of sexually dimorphic brain circuits. According to this model, exposure, or lack of exposure, to androgens and to their estrogen metabolites during a critical period of perinatal development leads to the differential organization of the neural pathways that underlie male- and female-specific behaviors. This perinatal hormonal activity permanently modifies the ability to manifest certain sexually dimorphic behaviors in adulthood. In turn, differential exposure to sex steroid hormones at puberty activates the pre-established male- and female- neural circuitry leading to sex-specific reproductive and social displays.

Sexually dimorphic brain areas

The discovery of the organizational effect of steroid hormones led to an extensive quest for target brain nuclei undergoing “masculinization” and “defeminization” in young males in order to generate the effective control centers of sex-specific behaviors such as mating and courtship in the adult [6].

Initial clues were obtained by Goy and Phoenix [7] who found that destruction of brain tissue lying in the midventral region of the hypothalamus resulted in a significant decrease in sexual receptivity in female guinea pigs. In male rats extensive injury to the neocortex results in a decreased tendency to copulate [8], whereas similar lesion in females produced no significant change in sexual behavior [9].

Surprisingly, in rodents, only a limited number of brain structures were found to exhibit clear sexual dimorphism despite the striking differences in behavior between the sexes. In rats these include the sexual dimorphic nucleus of the preoptic area (SDN-POA) and the posterodorsal medial amygdala (MePD), both of which are larger in males than females, and the anteroventral periventricular nucleus (AVPV), which is larger in females than in males [10–11]. Subtle sex differences in levels of neurotransmitters, in numbers of neurons projecting to specific brain nuclei, in shapes of synapses, and length of dendrites, were also detected in brain structures such as the medial amygdala, bed nuclei of the stria terminalis (BNST), medial preoptic area (MPOA) and AVPV [12–14].

Interestingly, most sexually dimorphic neural structures were found to express differential levels of gonadal steroid hormone receptors [15–17] and were described as components of the vomeronasal system, a sensory system thought to process pheromone information and required for sexual and aggressive responses [12–14].

The central role of the vomeronasal pathway

In rodents, pheromone detection is essential for triggering innate sexually dimorphic responses [reviewed in, 18–19]. For example, in males, female pheromones lead to testosterone and LH surges, puberty acceleration as well as initiation of immediate behavioral responses such as courtship display, ultrasound vocalizations and copulatory behavior. By contrast male pheromones can delay puberty onset of other males and promote conspecific male-male aggression and territoriality scent marking. In females, male pheromones can accelerate puberty onset (Vandenbergh effect), block pregnancy (Bruce effect), induce estrous cycle (Whitten effect) and sexual receptivity (lordosis behavior), while group-housed female pheromones lead to puberty delay and estrous cycle inhibition. Other female-typical behaviors such as maternal aggression, nesting and pup retrieval, were also shown to be mediated by pheromones.

From these results, the VNO emerged as a structure essential for pheromone detection, and responsible for the activation of innate sexually dimorphic reproductive behavioral and endocrine responses in males and females.

Recent studies have challenged some of the prevailing hypotheses described above, and have shed new light on mechanisms thought to underlie the organization and the activation of sexual dimorphic traits in vertebrates.

Gonadal sex hormones versus genomic information

In birds and mammals sexual differentiation is primary promoted by two mechanisms:

1. The differentiation of sex-specific gonads is determined by the function of sex chromosome genes. In mammals, the presence or absence of the testis-determining gene SRY, directs the undifferentiated gonad to form a testis that produces testicular hormones. The timing and level of SRY expression in males must occur in a narrow window during gonad development in order to activate the male pathway [20]. In absence of appropriate SRY expression, the default female pathway is activated.

2. Sex-specific traits in nongonadal tissues are determined by sex steroids (e.g. testosterone and estrogen) secreted from the developing gonads. The release of perinatal hormones promotes sex-specific differentiation of neuronal structures by affecting many cellular processes, such as gene expression, cell division and migration, neuronal growth, as well as apoptosis, and synapse formation [21–24]. As a result of steroid action, the network becomes sexually dimorphic with respect to the number and the types of neurons produced in each brain nucleus. This, in turn, influences the baseline number and types of sex-steroid receptors expressed in each brain area, as well as the strength of connectivity between various neuronal populations. In adulthood, steroid hormones act on sexually dimorphic brain regions to promote sex-specific behaviors.

In addition to these primary mechanisms, recent observations have documented the sexual differentiation of neuronal and non-neuronal structures occurring independently from gonadal instructions. This suggests the existence of cell-autonomous information unrelated to gonadal hormones that participates in the establishment of tissue sex differentiation.

In Tammar wallabies, the scrotum of males and the mammary primordia of females are already sexually differentiated on the day of birth, even though the gonads are still undifferentiated [25]. Moreover, neurons dissociated from embryonic male or female rat mesencephalon harvested before the onset of gonadal secretion and cultured under identical conditions nevertheless display sexually dimorphic numbers of TH-positive neurons, and of hypothalamic GABAergic and prolactin neurons. Importantly, these results are not affected by hormonal manipulations suggesting that they indeed originate from hormone independent mechanisms [26–29]. Furthermore, a significant number of transcripts are differentially expressed in early male and female mouse embryos (E10.5), before any detectable gonadal hormone influence [30].

The direct investigation of SRY-independent, gonadal-independent sex-specific events was made possible by the construction of transgenic mice in which the genetic sex of the brain (XX or XY) is made independent of the gonadal sex (testes or ovaries). The SRY gene was deleted or expressed in various genetic backgrounds: XY⁻ animals lack the SRY gene on the Y chromosome and are thus gonadally females but genetically males with an SRY deficiency; XXSry animals which express SRY under the control of an autosomal gene in a background of XX are gonadally males, and XY⁻Sry animals which express SRY under the control of an autosomal gene in a background of XY- are gonadally and genetically males.

Using these genetic tools, it was found that cultures consisting of XY⁻ or XY⁻Sry cells developed more TH- immunoreactive neurons than those derived from XX or XXSry cells [31], demonstrating the existence of SRY-independent sex-chromosome effects. Further, vasopressin-immunoreactive fibers in the lateral septum of XY⁻Sry males were more masculinized than XXSry males, and XY⁻ females were more masculinized than XX females. XY males also differed from XY⁻Sry males in several behavioral parameters such as in male-typical mating and sniffing behavior [32].

These results uncover the existence of X- and Y-specific genetic components underlying the development of sex-specific traits that are independent of SRY and gonad-derived sex hormones [33–34].

A particularly striking finding that lends further support to this hypothesis was recently found in the zebra finch songbird. In songbirds, males but not females, sing a courtship song. The brain nuclei involved in singing are significantly larger in males than in females. A rare bilateral gynandromorphic finch was analyzed in which the right half of the brain, body and sex organs were genetically and morphologically male, while the left structures were female (Figure 1). Remarkably, even though the two sides of the brain had been equally exposed to circulating gonadal hormones during development and adulthood, the neuronal song circuit on the right brain side appeared more masculine than that on the left (Figure 1), leading to behavioral singing phenotype that resembles a typical male [35]. This result provides further evidence that sex-specific behavioral traits rely on factors independent of circulating gonadal sex hormones and utilize cell autonomous information provided by sex chromosome genes.

Figure 1.

A bilateral gynandromorphic finch in which the right half of the body (A), gonads (B), sex-chromosome linked gene expression in the brain (C), and volume of sexually dimorphic nuclei (D) have male characteristics, while the left half is female-like. Striking sex differences have developed and are maintained in each side of the brain although they have been exposed equally to circulating gonadal hormones. This result supports the existence of cell-autonomous information provided by sex chromosome genes and independent from gonadal sex hormones (modified from Agate RJ, et al., 2003 with permission from authors).

Both vomeronasal and main olfactory systems mediate normal pheromone-evoked sexual behavior

Most mammals detect chemosensory signals through two distinct olfactory sensory epithelia; the main olfactory epithelium (MOE) which is located in the posterior recess of the nasal cavity, and the vomeronasal organ (VNO), a bilateral tubular shaped sensory structure located in the anteroventral part of the nasal septum.

Olfactory and vomeronasal sensory neurons express distinct types of chemosensory receptor genes and signal transduction machineries [36]. Moreover, each olfactory system uses a different pathway of segregated brain nuclei to process the associated chemosensory information. MOE neurons project to the main olfactory bulb and in turn to higher brain region including the olfactory cortex and the lateral amygdala. In contrast, VNO axons project to the accessory olfactory bulb and in turn to the medial and basolateral amygdala, including the BNST and MeA, and then to specific nuclei of the hypothalamus such as the MPOA and the ventromedial hypothalamus.

Classical studies based on chemical and surgical ablations of the VNO and the MOE had suggested distinct roles for the olfactory and vomeronasal systems in the detection and processing of chemosensory information. From these earlier studies, the main olfactory system appeared essential for the detection of small volatile odors, leading to the cognitive and emotional responses to smell. In contrast, the vomeronasal system appeared dedicated to the detection of non-volatile pheromone cues that elicit mating behavior and sex-specific aggression. However, the assumption of a simple functional dichotomy between the olfactory and vomeronasal systems was recently contradicted by data obtained from animals in which VNO or MOE function was genetically impaired. In particular, these studies provided direct evidence that the main olfactory system plays a major role in pheromone-evoked behavior and endocrine innate responses, while the vomeronasal system ensures the sex-specificity of the behavioral responses.

Genetic ablation of the TRPC2 channel, a signaling molecule essential for VNO-mediated pheromone signaling, leads to indiscriminate courtship and mounting behavior of TRPC2^−/− male mice towards both males and females, and to the absence of pheromone-evoked aggression towards male intruders [37–38]. Surprisingly, the TRPC2^−/− male mice do not show any impairment in mating behavior with females and instead, display high level of courtship and copulatory behaviors indistinguishable from those observed in normal wild-type males. These results provided direct evidence that VNO stimulation is not necessary for triggering male-female mating behavior, although it is required for male-male aggression and sex discrimination. Recent work has identified the major urinary proteins as key components of pheromonal signals that promote male-male aggression through the vomeronasal system [39**]

In a different set of experiments, genetic ablation of signal transduction components of MOE neurons (OCNC/Cnga2 and AC3) lead to a severe reduction in male mating behavior [40, 41*, 42**]. Moreover, chemical lesions of the MOE lead to a similarly striking reduction of male reproductive behavior [41**] and female sexual receptivity [43].

The direct involvement of the main olfactory system in pheromone detection was further supported by experiments in which subpopulations of MOB mitral cells were shown to be activated by MTMT, a pheromone compound of male mouse urine that attracts females [44**]. Similarly, the activity of 2-methybut-2-enal (2MB2), a pheromone secreted in female rabbit milk that elicits reaching and oral grasping of maternal nipples in newborn pups [45] is also mediated by the main olfactory system. Furthermore, trace amine-associated receptors (TAARs), shown to be activated by urine compounds with pheromonal activity were identified as a new class of olfactory receptors expressed in the mouse olfactory epithelium [46*].

In further support of the critical role of the main olfactory system in mediating pheromone-evoked behaviors and endocrine changes, two independent genetic neuronal tracing approaches have demonstrated that neurons of the hypothalamus that synthesize LHRH, a key neurohormone controlling reproduction and fertility, receive major afferent sensory inputs from the main olfactory system [42**,47**]. Notably, the study by Yoon et al. [42**] revealed that, in contrast to the established notion that LHRH release is regulated by vomeronasal inputs, LHRH neurons fail to receive synaptic inputs from the vomeronasal system.

The mouse brain is functionally bisexual

Sex-specific behaviors are generally assumed to be under the tight control of gonadal hormones and other factors that organize and activate sex-specific neuronal circuits. The commonly accepted concept of major anatomical and functional differences between the male and the female brains (Figure 2A) has been at least partially questioned in the past by studies demonstrating that adult males and females are capable of displaying portions of behaviors characteristic of the other sex. In many species, females may, under certain conditions and without hormonal manipulation, mount other females in a manner similar to the copulatory pattern of males of the same species. These male-like displays from females have been observed in many different mammalian species including, rats, domestic cattle, carnivores and even in primates [review in 48–49]. Similarly male rodents have been shown to display female-typical lordosis [50]. These observations have led several researchers including Beach [50–51] and Newmann, [52] to propose that the brains of both sexes may contain the neuronal template required to carry out portions of the behaviors of the opposite sex.

Figure 2.

According to the prevailing dogma of brain sexual differentiation, gonadal sex-hormones organize and activate hard-wired neural circuits mediating sex-specific behaviors that are distinct in the male and the female brains (A). Recent findings in the mouse suggest of a new model according to which fully functional male- and female-behavior circuits develop and are maintained in the brain of both sexes and are under the control of a sex-specific vomeronasal sensory switch (B). By adopting discrete modules of circuits specific for the opposite sex, new behavioral traits might emerge during evolution that are exhibited by the opposite sex in closely related species. One possible example of such evolution is the parental behavior of male prairie voles, which is not displayed by males of other vole species (C). Furthermore, such network organization might account for the rapid behavioral sex-reversals, as occurs in many coral-reef fishes (D) (modified from Kimchi T, et al., 2007).

More recently, the behavioral analysis of female mice in which the vomeronasal organ had been surgically (VNOx) or genetically (TRPC2^−/−) ablated revealed a complete reversal of sex-specific behavior in females, demonstrating that, in fact, a fully functional male-behavior effector circuit exists in the brain of female mice that is normally repressed by the vomeronasal system. Accordingly, VNO-deficient female mice attempt to copulate with other mice in a male-specific manner and emit male-specific ultrasound courtship vocalizations when put in presence of a female or a male intruder mouse. Furthermore, these females display solicitation behavior, including intense sniffing and sexual chasing of an intruder mouse followed by mounting and pelvic thrusting similar to the behavior exhibited by control wild type males toward intruder females [53**]. Side-by-side comparison of the male-like sexual and courtship behaviors displayed by females TRPC2^−/− and wild-type males show no statistical difference. In addition, females with impaired VNO function display severe defects in maternal behavior.

Surprisingly, although TRPC2^−/− and VNOx females behave similarly to males, their estrous cycle, body weight and breeding success parameters were that of normal wild-type females. Moreover, radioimmunoassays for sex hormone levels in the serum of mutant females were similar to those of wild-type females.

These findings demonstrate that an entirely functional neural effector network mediating male-specific reproductive behavior develops and is maintained throughout adulthood in the female brain (Figure 2B). Furthermore, it indicates that VNO inputs act to repress the initiation and maintenance of male-typical behavior in females.

It is tempting to speculate that, similarly, VNO-inputs might act in males to repress female-typical behaviors such as maternal nursing behavior and lordosis sexual behavior. Interestingly, and in further support of this hypothesis, several studies in rats have documented the role of VNO-mediated pheromone signals in naïve males for repression of female-specific responses [54]. For example, bilateral lesion of the AOB appears to facilitate the expression of lordosis behavior in hormone-manipulated naive males [55–56]. Furthermore, while virgin males are more likely to kill newborn pups than females, this infanticide behavior is significantly reduced in VNOx males, which appear more likely to exhibit maternal behavior including nest building, pup retrieval and nursing posture [57–58].

Taken together, these findings suggest that in both male and female rodents, pheromonal inputs that are detected and processed through the vomeronasal system act to tonically inhibit the expression of social and reproductive innate responses typically shown by the opposite sex. Elimination of this pheromone-mediated sensory repression during development or adulthood results in the rapid reversal of sex-specific behavioral repertoires, independently from the organizational action of gonadal hormones during development.

Evolutionary considerations

The existence of male and female-specific effector circuits in the brain of each sex may provide a flexible behavioral system that enables the rapid evolution of behavioral traits in closely related species (Figure 2C). For example, the well noted parental care displayed by the male prairie vole, which contrasts with the lack of male-pup interactions in the closely related mountain vole [59], can be seen as a minor modification of sex-specific behavioral networks, such that the microcircuit underlying maternal behavior is now also part of the male behavioral repertoire of the prairie vole (Figure 2C). Such dramatic behavioral differences in closely related species would be difficult to account for if male and female behavior circuits were anatomically present only in the brain of the corresponding sex.

Further examples of simple behavioral switches that triggers rapid changes from male- to female-specific behaviors can be found in lower vertebrates, with particularly striking examples in parthenogenetic lizards and hermaphroditic fish (Figure 2D).

Courtship and mating behavior in the whiptail lizards of the genus Cnemidophorus is highly ritualized [60]. Within this genus, some species are parthenogenetic, such that all members of the species are triploid females that reproduce by cloning, without any requirement for males. These unisexual female lizards exhibit behaviors remarkably similar to the courtship and copulatory behaviors of gonochoristic species (i.e. both females and males reproduce sexually). species. One female displays male-like courtship and mounting while the receptive female demonstrates female-typical behavior. Two weeks later, these roles reverse and the mounting individual is now mounted. Importantly, androgen levels in parthogenotes exhibiting male-like pseudosexual behavior are similar to that of parthogenotes with female-like behaviors, and to that of the gonochoristic females.

At the level of the brain, the preoptic area (POA) of the gonochoristic species is larger in males and is involved in male-typical mounting behavior, whereas the ventromedial hypothalamus (VMH) is larger in females and controls female sexual receptivity. Remarkably, the POA and VMH in parthenogenetic lizards are similar in size to that of a typical female, even in individuals exhibiting male-like pseudosexual behavior. Moreover, 2DG uptake, a measure of metabolic activity in the brain, shows that metabolic activity is higher in the POA of parthogenetic female lizards exhibiting male-like behaviors and is higher in the VMH of females with female-like pseudosexual behavior [61].

These data strongly imply that in reptiles, as in the mouse, the brain of each sex contains the neuronal components required to produce the full behavioral repertoire of the opposite sex.

Other striking examples of behavioral switches between sexes can be found in various species of fish. In the haremic coral-fish Labrodes dimidiatus, the disappearance of the male from the harem induces the largest female to exhibit male-typical sexual behavior, and to reverse back to female behavior in the range of an hour [62]. Other species of coral fishes are capable of switching from female to male behaviorally and morphologically, including gonad alteration. Remarkably, some species, such as the Okinawa goby fishes, can undergo behavioral and morphological sex-reversal in both directions multiple times [63]. In general, changes in the sex-specificity of behavior are extremely rapid compared to the development of functional testes that takes days to weeks [64]. These results suggest that rapid brain changes that affect sex-typical social and reproductive behaviors are not triggered by change in gonad sex hormones [65] and that, as noted by Kobayashi et al [66], the fish brain is fundamentally bisexual.

Conclusion and future direction

A long-standing and fundamental question that has intrigued biologists, psychologists, sociologists and many others scientists is ‘how do sex differences in behavior arise?’

Rodents exhibit a wide range of sex-specific innate social and reproductive responses, providing an attractive animal model to uncover mechanisms underlying sexual dimorphism of brain function in mammals.

So far, the central prevailing dogma of brain sexual differentiation has proposed that gonadal sex-hormones secreted early in life organize hard-wired sex-specific neural circuits that mediate sex-specific behaviors. However, new advances in molecular and genetic strategies demonstrate that gonadal hormones are not the sole factor responsible for the emergence of sex-specific brain function. Instead, sex chromosome genes, as well as constitutive chemosensory information, seem to have a profound influence on the organization of male- and female- specific social and sexual displays.

Many basic questions are still to be addressed: what are the mechanisms by which pheromone inputs can alter the function of sex-specific neuronal pathways; what is the relationship between the effect of sex steroid hormones and sensory inputs on the organization of sexually dimorphic circuits controlling innate reproductive responses; and what are the pheromone receptors and ligands required to establish sex-specific responses? Further, establishing the role of sensory information in the modulation of sex-specific neuronal processing in other animal species and in humans remains an intriguing challenge. Finally, insights on mechanisms underlying the organization and function of normal male and female brain in species ranging from fish to reptiles and mammals illustrate the power of studies performed in ethologically relevant social and environmental contexts and that avoid extensive hormonal manipulations.