Serotonin and arginine–vasopressin mediate sex differences in the regulation of dominance and aggression by the social brain

Significance

There are profound sex differences in the expression of social behavior and in the incidence of many psychiatric disorders, and yet little is known about how the brain mechanisms underlying these phenomena differ in females and males. Here, we report that serotonin (5-HT) and arginine–vasopressin (AVP) act in opposite ways within the hypothalamus to regulate dominance and aggression in females and males. Dominance and aggression are promoted by 5-HT in females and by AVP in males. Because dominance and aggressiveness have been linked to the resistance to stress-related psychiatric disorders, these disorders may be more effectively treated with 5-HT–targeted drugs in females and AVP-targeted drugs in males.

Abstract

There are profound sex differences in the incidence of many psychiatric disorders. Although these disorders are frequently linked to social stress and to deficits in social engagement, little is known about sex differences in the neural mechanisms that underlie these phenomena. Phenotypes characterized by dominance, competitive aggression, and active coping strategies appear to be more resilient to psychiatric disorders such as posttraumatic stress disorder (PTSD) compared with those characterized by subordinate status and the lack of aggressiveness. Here, we report that serotonin (5-HT) and arginine–vasopressin (AVP) act in opposite ways in the hypothalamus to regulate dominance and aggression in females and males. Hypothalamic injection of a 5-HT1a agonist stimulated aggression in female hamsters and inhibited aggression in males, whereas injection of AVP inhibited aggression in females and stimulated aggression in males. Striking sex differences were also identified in the neural mechanisms regulating dominance. Acquisition of dominance was associated with activation of 5-HT neurons within the dorsal raphe in females and activation of hypothalamic AVP neurons in males. These data strongly indicate that there are fundamental sex differences in the neural regulation of dominance and aggression. Further, because systemically administered fluoxetine increased aggression in females and substantially reduced aggression in males, there may be substantial gender differences in the clinical efficacy of commonly prescribed 5-HT–active drugs such as selective 5-HT reuptake inhibitors. These data suggest that the treatment of psychiatric disorders such as PTSD may be more effective with the use of 5-HT–targeted drugs in females and AVP-targeted drugs in males.

Prominent sex differences occur in the incidence, development, and clinical course of many psychiatric disorders. Women, for example, have higher rates of depression and anxiety disorders such as posttraumatic stress disorder (PTSD), whereas men more frequently suffer from autism and attention deficit disorders (1–4). Because little is known about sex differences in the efficacy of treatments for these disorders, current treatment strategies are largely the same for both sexes. The development of effective treatments for both women and men can proceed with a clear understanding of sex differences in the mechanisms and etiology of psychiatric disorders. Many of these disorders are linked to deficits in adaptive social skills (5, 6); therefore, understanding the neural mechanisms underlying social engagement in both sexes is essential. In most mammalian species, social interactions among both sexes are governed by dominance relationships. As such, behaviors associated with these relationships (e.g., social recognition, stress, competitive aggression) are the foundation for social interactions and are highly relevant for understanding psychiatric disorders (7–9). Emerging genetic and environmental evidence suggests that traits such as dominance, competitive aggression, and active coping strategies are linked, resulting in phenotypes more resistant to psychiatric disorders (10–12). In contrast, subordinate status and the absence of aggression are linked to phenotypes more susceptible to adverse psychiatric outcomes (e.g., PTSD). Despite the importance of understanding the basic mechanisms underlying aggression and dominance and their role in psychiatric disorders, possible sex differences in the basic neural mechanisms underlying these phenomena have received almost no attention.

Given that social behavior is evolutionarily ancient and social strategies used by females and males evolved in response to very different selective pressures, it seems likely that there are fundamental sex differences in the neural mechanisms regulating expression of social behaviors such as dominance and competitive aggression. And yet little is known about the neurobiology of dominance and competitive aggression, particularly in females. Studies of aggression in female rodents have focused almost exclusively on maternal aggression at least in part because female laboratory rodents rarely display spontaneous aggression. In contrast, Syrian hamsters provide an outstanding rodent model with which to investigate competitive strategies in females as well as males because aggression does not have to be induced as in many other social species (e.g., mice and rats) by artificial means (e.g., electric shock) or by mating-induced pair bonding (e.g., voles) (13–15). Female hamsters, like female monkeys, display a range of competitive strategies including the rapid formation of robust hierarchal dominance relationships and the ability to inhibit the reproductive capacity of other females (9, 16). Both female and male hamsters also readily display a variety of social behaviors that are fundamental for social relationships. In addition, a great deal is known about the biological mechanisms controlling social behavior in Syrian hamsters, making it a powerful model system for preclinical study of behaviors that underlie psychiatric health and illness (9, 17).

One of the most well-known phenomena in behavioral neuroscience is the ability of serotonin (5-HT) to inhibit impulsive behaviors, including aggression, in males in species ranging from invertebrates to primates (18–20). In contrast to 5-HT, centrally administered arginine–vasopressin (AVP) stimulates aggression in males, and antagonists of AVP V1a receptors (V1aRs) reduce aggression (13, 14, 21). In support of the hypothesis that there are fundamental sex differences in the neural mechanisms regulating social behavior, we recently found that AVP has the opposite effect on competitive aggression in female hamsters (22). In contrast to males, central administration of AVP in females reduces aggression, and V1aR antagonists stimulate aggression. Remarkably, despite hundreds of studies demonstrating that 5-HT inhibits aggression in males, the effects of 5-HT on aggression in females remain essentially unknown (23, 24).

In the following experiments, we investigated the hypothesis that there are fundamental sex differences in the neural regulation of aggression and dominance. More specifically, we investigated whether 5-HT promotes and AVP inhibits aggression and dominance in females and whether 5-HT inhibits and AVP promotes aggression and dominance in males. These studies support this hypothesis by directly comparing sex-specific effects of 5-HT and AVP on aggression and by demonstrating that acquisition of dominance is associated with activation of raphe 5-HT neurons in females and hypothalamic AVP neurons in males. Together, these data provide strong support for the hypothesis that there are striking sex differences in the mechanisms regulating social behavior. Given the emerging relationship of aggression and dominance with phenotypes that are resilient to psychiatric disorders, understanding how the 5-HT and AVP systems control aggression and dominance has the potential for significant translational impact. For example, the following experiments demonstrate that systemic administration of one of the most commonly prescribed selective 5-HT reuptake inhibitors (SSRIs) (i.e., fluoxetine) has opposite effects in females and males, suggesting that clinical efficacy of SSRIs may differ dramatically between the sexes.

Despite widespread use of SSRIs, there appear to be no studies examining gender differences in the efficacy of SSRIs to treat stress disorders and only a limited number examining SSRI efficacy to treat depression. Of the published peer-reviewed studies, five found no difference in the efficacy of SSRIs in men and women (25–29), and six found SSRIs to be more effective in women (30–35). There are no reports of SSRIs being more effective in men. Thus, these data do not generate confidence that there is comparable clinical efficacy of SSRIs in men and women but rather suggest that SSRIs are more effective as an antidepressant in women than in men. The lack of definitive data is remarkable but not surprising. It is not uncommon that large sex differences in the efficacy of therapies used by millions are overlooked in clinical practice (36, 37).

Results

Activation of 5-HT1aRs in the AH Stimulates Aggression in Females.

In view of the substantial evidence that activation of 5-HT1a receptors (5-HT1aRs) inhibits aggression in males, we examined the effects of the 5-HT1aR agonist 7-(Dipropylamino)-5,6,7,8-tetrahydronaphthalen-1-ol (8-OH-DPAT) in females (24). Microinjection of 8-OH-DPAT into the anterior hypothalamus (AH) (Fig. S1; see ref. 38) produced a dose-dependent increase in aggression, F(3, 27) = 4.36, P < 0.05 (Fig. 1A), and a decrease in latency to attack, F(3, 26) = 3.27, P < 0.01 (Fig. S2A). Five hundred micromolar and 1 mM 8-OH-DPAT increased the duration of aggression (P < 0.05 and P < 0.01, respectively; Fig. 1A) and decreased the latency to attack (P < 0.05; Fig. S2A) compared with controls. Because of the surprising finding that 8-OH-DPAT stimulated aggression in females, we directly compared the effects of 8-OH-DPAT in females and males. There was a significant interaction between sex and drug treatment on the duration of aggression, F(1, 27) = 37.50, P < 0.01 (Fig. 1B). 8-OH-DPAT treatment increased the duration of aggression in females, t(15) = 3.46, P < 0.01, and decreased the duration of aggression in males, t(13) = 6.82, P < 0.01. There was a significant interaction between drug treatment and sex on latency to attack, F(1, 28) = 16.24, P < 0.01 (Fig. S2B). Although 8-OH-DPAT treatment did not affect latency to attack in females, t(15) = 1.55, P > 0.05, it increased latency to attack in males, t(13) = 3.99, P < 0.01.

Fig. S1.

Sites of drug injections into the AH at (A) –0.3 mm, (B) –0.6 mm, and (C) –1.2 mm from bregma as determined by postmortem histological analysis. Closed circles are considered “hits” and open circles “misses” (figure modified from ref. 38).

Fig. 1.

Sex differences in the effects of 8-OH-DPAT on aggression. (A) Dose-dependent effects of 8-OH-DPAT injected into the AH of female hamsters on the duration of aggression. (B) Direct comparisons between females and males of the effect of 8-OH-DPAT (1 mM and 100 µM, respectively) on duration of aggression. Error bars indicate SEM. *P < 0.05; ***P < 0.01.

Fig. S2.

Sex differences in the effects of 8-OH-DPAT on attack latency. (A) Dose-dependent effects of 8-OH-DPAT injected into the AH of female hamsters on attack latency. (B) Direct comparisons between females and males of the effect of 8-OH-DPAT (1 mM for females and 100 µM for males) on attack latency. Error bars indicate SEM. *P < 0.05; ***P < 0.01.

8-OH-DPAT and AVP Alter the Expression of Aggression in Males and Females in Opposite Ways.

Next, we directly compared the effects of 8-OH-DPAT, AVP, and 8-OH-DPAT combined with AVP (8-OH-DPAT/AVP) on aggression following injection into the AH of females and males. There was a significant interaction between sex and drug treatment on the duration of aggression, F(3,48) = 34.14, P < 0.01 (Fig. 2). 8-OH-DPAT–treated females were more aggressive than control females (P < 0.01), whereas AVP-treated females were less aggressive than control females (P < 0.05; Fig. 2A). 8-OH-DPAT/AVP–treated females were more aggressive than AVP-treated females (P < 0.05) and less aggressive than 8-OH-DPAT–treated females (P < 0.05) but did not differ from control females (P > 0.05; Fig. 2A). 8-OH-DPAT–treated males were less aggressive than control males (P < 0.01), whereas AVP-treated males were more aggressive than control males (P < 0.05; Fig. 2A). 8-OH-DPAT/AVP–treated males were less aggressive than AVP-treated males (P < 0.01) and control males (P < 0.01), whereas 8-OH-DPAT/AVP–treated males were more aggressive than 8-OH-DPAT–treated males (P < 0.05; Fig. 2A). Direct comparisons between females and males revealed that 8-OH-DPAT–treated females were more aggressive than 8-OH-DPAT–treated males, t(13) = 10.14, P < 0.01 (Fig. 2B). 8-OH-DPAT/AVP–treated females were more aggressive than 8-OH-DPAT/AVP–treated males, t(8) = 3.58, P < 0.01 (Fig. 2B). AVP-treated females were less aggressive than AVP-treated males, t(12) = 7.03, P < 0.01 (Fig. 2B). There was a significant interaction between sex and drug treatment for attack latency, F(3, 49) = 12.58, P < 0.01 (Fig. S3). AVP-treated females had a longer latency to attack than 8-OH-DPAT–treated females (P < 0.05) and a strong trend compared with 8-OH-DPAT/AVP–treated females (P = 0.06; Fig. S3A). 8-OH-DPAT–treated males had a longer latency to attack than all other male groups (P < 0.01; Fig. S3A). Direct comparisons between females and males revealed that AVP-treated females had a longer latency to attack than AVP-treated males, t(12) = 3.84, P < 0.01 (Fig. S3B). 8-OH-DPAT–treated females had a shorter latency to attack than 8-OH-DPAT–treated males, t(13) = 5.84, P < 0.01 (Fig. S3B).

Fig. 2.

Effects of 8-OH-DPAT (DPAT), AVP, and their combined injection into the AH on aggression in females and males. Drug concentrations used were determined by dose–response studies (Fig. 1, Fig. S2, and ref. 50). Duration of aggression was compared between females and males that received AVP (0.9 µM), vehicle, 8-OH-DPAT/AVP mixture (1 mM and 0.9 µM, respectively), or 8-OH-DPAT (1 mM for females and 100 µM for males). Female N: AVP = 8; vehicle = 10; DPAT/AVP = 4; DPAT = 7. Male N: AVP = 6; vehicle = 7; DPAT/AVP = 6; DPAT = 8. (A) Within-sex comparisons of drug treatments on duration of aggression. (B) Direct comparisons of drug treatments on duration of aggression between females and males. Error bars indicate SEM. *P < 0.05; ***P < 0.01.

Fig. S3.

Effects of 8-OH-DPAT (DPAT), AVP, and their combined injection into the AH on attack latency in females and males. Drug concentrations used were determined by dose–response studies (Fig. 1, Fig. S2, and ref. 50). Attack latency was compared between females and males that received AVP (0.9 µM), vehicle, a mixture containing 8-OH-DPAT/AVP (1 mM and 0.9 µM, respectively), or 8-OH-DPAT (1 mM for females and 100 µM for males). (A) Within-sex comparisons of different drug treatments on attack latency. (B) Direct comparisons of drug treatment on attack latency between females and males. Error bars indicate SEM. *P < 0.05; ***P < 0.01.

Systemically Administered Fluoxetine Alters Aggression in Opposite Ways in Males and Females.

We tested the hypothesis that systemically administered fluoxetine increases aggression in females. There was a dose-dependent increase in duration of aggression, F(2, 34) = 3.70, P < 0.05 (Fig. 3A). Treatment with 20 mg/kg fluoxetine increased aggression compared with vehicle (P < 0.05). There was also a dose-dependent decrease in the latency to attack, F(2, 34) = 3.65, P < 0.05 (Fig. S4A). Treatment with 20 mg/kg fluoxetine significantly decreased attack latency compared with vehicle (P < 0.05). We directly compared the effect of fluoxetine on aggression in females and males. There was an interaction between fluoxetine treatment and sex on the duration of aggression, F(1, 37) = 47.62, P < 0.01 (Fig. 3B). A dose of 20 mg/kg fluoxetine increased the duration of aggression in females, t(29) = 2.57, P < 0.05, and decreased the duration of aggression in males, t(8) = 5.66, P < 0.01. There was an interaction between fluoxetine treatment and sex on latency to attack, F(1, 37) = 35.77, P < 0.01 (Fig. S4B). A dose of 20 mg/kg fluoxetine decreased latency to attack in females, t(29) = 2.63, P < 0.05, and increased latency to attack in males, t(8) = 4.160, P < 0.01.

Fig. 3.

Sex differences in the effects of fluoxetine on aggression. (A) Dose–response study of the effects of systemically injected fluoxetine on duration of aggression in females. (B) Direct comparisons between females and males of the effects of fluoxetine on duration of aggression. Error bars indicate SEM. *P < 0.05; ***P < 0.01.

Fig. S4.

Sex differences in the effects of fluoxetine on attack latency. (A) Dose–response study of the effects of systemically injected fluoxetine on attack latency in female hamsters. (B) Direct comparisons between females and males of the effects of fluoxetine on attack latency. Error bars indicate SEM. *P < 0.05; ***P < 0.01.

Dominance and the Activation of 5-HT Neurons.

We hypothesized that activity of 5-HT cells in the raphe, as measured by colocalization of 5-HT-immunoreactivity (ir) and fos-ir, is up-regulated by acquisition of dominance in female hamsters. In the anterior dorsal raphe nucleus (DRNa) (Fig. 4 A and B), there was a significant effect of dominance status, F(2, 54) = 3.081, P = 0.05, but not sex, F(1, 54) = 0.686, P > 0.05, nor an interaction, F(2, 52) = 1.311, P > 0.05, in 5-HT-ir/fos-ir. In the posterior DRN (DRNp) (Fig. 4 C and D), there was a significant effect of dominance status, F(2, 50) = 3.084, P = 0.05, and a trend toward an effect of sex, F(1, 50) = 3.165, P = 0.08, but no interaction, F(2, 50) = 0.383, P > 0.05, in 5-HT-ir/fos-ir. Within-sex a priori comparisons revealed that dominant females had more 5-HT-ir/fos-ir in the DRNa and DRNp than subordinate females (P < 0.05; Fig. 4 B and D). Dominant females had more 5-HT-ir/fos-ir in the DRNa and a trend toward more 5-HT-ir/fos-ir in the DRNp than control females (P < 0.05 and P = 0.08, respectively; Fig. 4 B and D). There was no effect of dominance status on 5-HT-ir/fos-ir in the DRNa and DRNp of males (P > 0.05; Fig. 4 B and D). In the median raphe nucleus (MRN) (Fig. 4 E and F), there was no effect of dominance status, F(2, 48) = 1.736, P > 0.05, but a strong trend toward an effect of sex, F(1, 48) = 3.758, P = 0.06, and no interaction, F(2, 48) = 1.648, P > 0.05, in 5-HT-ir/fos-ir. Within-sex a priori comparisons revealed no differences in 5-HT-ir/fos-ir for either females or males (P > 0.05; Fig. 4F).

Fig. 4.

Immunofluorescent colocalization of 5-HT-ir (teal) and fos-ir (red) in cells within regions of the raphe. DRNa (A and B), DRNp (C and D), MRN (E and F), vrDRNa (G and H), and drDRNa (I and J). Magnification is indicated in the lower left corner. Yellow boxes indicate subregions magnified to 40×. (Scale bars, 20 µM.) Purple boxes in G and I represent region boundaries for the vrDRNa and drDRNA, where 5-HT-ir/fos-ir cells were quantified. Graphs indicate the percentage of 5-HT-ir cells that colocalize fos-ir (percentage of activated 5-HT cells) as a function of dominance status and sex in DRNa, DRNp, MRN, vrDRNa, or drDRNa. Error bars indicate SEM. *P < 0.05.

We also examined whether dominance status altered 5-HT-ir/fos-ir in the ventral (vrDRNa) and dorsal (drDRNa) subdivisions of the most rostral portion of the DRNa in female and male hamsters because a previous study in male hamsters found subordinates have significantly more 5-HT-ir/fos-ir in the vrDRNa than dominant and control hamsters (17). In the vrDRNa (Fig. 4 G and H), there was an effect of dominance status on 5-HT-ir/fos-ir in males, F(2, 23) = 3.902, P < 0.05, with subordinate males having more 5-HT-ir/fos-ir compared with dominants and controls (P < 0.05; Fig. 4H). In females, there no effect of dominance status on 5-HT-ir/fos-ir in the vrDRNa, F(2, 26) = 0.723, P > 0.05, nor were there differences between dominant, subordinate, and control females (P > 0.05; Fig. 4H). In the drDRNa (Fig. 4 I and J), there was an effect of dominance status on 5-HT-ir/fos-ir in males, F(2, 23) = 6.355, P < 0.01, with dominant males having more 5-HT-ir/fos-ir than controls or subordinates (P < 0.05 and P < 0.01, respectively; Fig. 4J). In females, there was an effect of dominance status on the number of 5-HT-ir/fos-ir cells in the drDRNa, F(2, 26) = 6.431, P < 0.01. Dominant females had more 5-HT-ir/fos-ir than controls and subordinates (P < 0.01 and P < 0.05, respectively; Fig. 4J).

Dominance and the Activation of AVP Neurons.

In males, we hypothesized that the activity of AVP-containing cells, as measured by the colocalization of AVP-ir/fos-ir, is up-regulated in hypothalamic nuclei by the acquisition of dominance. In the medial supraoptic nucleus (mSON) (Fig. 5 A and B), there was an effect of dominance status on AVP-ir/fos-ir, F(1, 55) = 31.51, P < 0.01, but no effect of sex, F(1, 55) = 0.32, P > 0.05, and no interaction, F(2, 55) = 1.37, P > 0.05. Within-sex a priori comparisons revealed subordinate males had more AVP-ir/fos-ir than control males (P < 0.01) and dominant males had more AVP-ir/fos-ir than subordinate males (P < 0.05) and control males (P < 0.01; Fig. 5B). In females, both subordinate and dominant animals had more AVP-ir/fos-ir than control females (P < 0.01), but there was no difference between dominants and subordinates (P > 0.05; Fig. 5B). In the nucleus circularis (NC) (Fig. 5 C and D), there was no effect of sex, F(1, 46) = 1.69, P > 0.05, but a main effect of dominance status, F(1, 46) = 15.75, P < 0.01, and a trend toward an interaction, F(2, 46) = 2.81, P = 0.07. Within-sex a priori comparisons revealed that dominant and subordinate females had more AVP-ir/fos-ir than control females (P < 0.01; Fig. 5D). Dominant males had more AVP-ir/fos-ir than control males (P < 0.01) and subordinate males (P < 0.01; Fig. 5D). In the medial paraventricular nucleus (PVN) (Fig. 5 E and F), there was an effect of dominance status, F(2, 51) = 13.46, P < 0.01; sex, F(1, 51) = 14.91, P < 0.01; and a trend toward an interaction, F(2, 51) = 2.70, P = 0.07. Within-sex a priori comparisons in females revealed no effect of dominance status on AVP-ir/fos-ir (P > 0.05; Fig. 5F). In males, both subordinate and dominant hamsters (P < 0.01; Fig. 5F) had significantly more AVP-ir/fos-ir than controls.

Fig. 5.

Immunofluorescent colocalization of AVP-ir (green) and fos-ir (red) in cells within hypothalamic regions. The mSON (A and B), NC (C and D), and medial PVN (E and F). Magnification is indicated in the lower left corner. Yellow boxes indicate subregions magnified to 40×. (Scale bars, 20 µM.) Graphs indicate the percentage of AVP-ir cells that colocalize fos-ir (percentage of activated AVP cells) as a function of dominance status and sex in mSON, NC, or PVN. Error bars indicate SEM. *P < 0.05; ***P < 0.01.

Discussion

These data support the hypothesis that there are fundamental sex differences in the neural regulation of aggression and dominance. Direct comparisons of 5-HT and AVP indicate that these neurochemical signals act in opposite ways within the same brain site to regulate aggression and dominance in females and males. Hypothalamic injection of a 5-HT-1a agonist stimulated aggression in females and inhibited aggression in males, whereas injection of AVP inhibited aggression in females and stimulated aggression in males. These data also provide evidence that there are striking differences in the neural mechanisms regulating the acquisition of dominance in females and males. Acquisition of dominance was associated with activation of 5-HT cell bodies within the DRN of females and activation of hypothalamic AVP neurons in males. Finally, systemic administration of fluoxetine increased aggression in females and substantially decreased aggression in males, suggesting that sex differences may be a critical factor in the clinical efficacy of 5-HT–active drugs.

The present data reinforce previous work indicating that the AH is a critical element within the neural circuitry controlling agonistic behavior (15, 39–41). Not only do 5-HT and AVP fibers project to this region (39, 42), the present data demonstrate that activation of 5-HT and AVP cell bodies are associated with acquisition of dominance in a sex-dependent manner. Dominant females displayed significantly more activation of 5-HT cell bodies within the DRN than subordinate or control females. In contrast, there were no significant differences in the number of 5-HT–activated cell bodies in the DRN among dominant, subordinate, or control males. Despite these dramatic sex differences, more subtle effects of dominance status were observed in subregions of the DRNa in both females and males. Interestingly, subordinate males, but not females, displayed significantly more activation of 5-HT cell bodies in the vrDRNa than dominant or control males, as has been reported previously, and dominance status altered the activation of 5-HT cell bodies in the drDRNa in both females and males (17). Therefore, although activation of 5-HT cell bodies throughout the DRN is associated with the acquisition of dominance in females and not males, more subtle changes within DRNa subregions can be related to dominance status in males as well as females.

In males, acquisition of dominance was strongly associated with activation of AVP cell bodies in the mSON and NC. Dominant males displayed significantly more activation of AVP cell bodies within both the mSON and NC than did subordinate or control males. In contrast, in females, a significantly larger number of AVP cell bodies were activated in the mSON and NC in both dominant and subordinate females than in controls. Thus, there is a substantial sex difference in the relationship between dominance and activation of AVP cell bodies in the mSON and NC. In males, activation of AVP cell bodies in these sites was associated with dominance, whereas in females activation of these neurons was independent of dominance status but was associated with social interaction. Interestingly, in the PVN, activation of AVP cell bodies in males was associated with social interaction, whereas no such relationship was observed in females.

Studies of the neural mechanisms underlying competitive aggression have been conducted almost exclusively in males, probably because of the longstanding emphasis on male–male competition in intrasexual selection (43). The present findings are consistent with the large body of previous work on the roles of 5-HT and AVP in regulating competitive aggression in males. In contrast, studies of the neural mechanisms underlying aggression in females have focused almost exclusively on maternal aggression (44). Although the importance of female competitive behaviors in achieving reproductive benefits has long been recognized, particularly in primates, little attention has been paid to their underlying neural mechanisms (16, 45). The present data demonstrate that although the behavioral expression of aggression and dominance in females and males is quite similar, 5-HT and AVP act in opposite ways within the same brain site to regulate these behaviors.

The strong but opposite relationship between activation of the 5-HT and AVP systems and the promotion of aggression/dominance in females and males may have significant clinical importance. For example, because dominance imparts a resistance to the adverse consequences of social stress, the findings of major sex differences in the neurochemical regulation of aggression/dominance may be important for our understanding of sex differences in the incidence and treatment of stress-related psychiatric disorders (12, 46). Indeed, it is possible that 5-HT–active drugs are more efficacious in treating some psychiatric disorders in females and AVP-active drugs are more efficacious in males. One of the most commonly prescribed 5-HT–active drugs for treatment of psychiatric disorders is fluoxetine (Prozac), and the possibility that there are sex differences in its efficacy is reinforced by our findings that the systemic administration of fluoxetine increases aggression in females and reduces it in males. In support of this possibility, systemic administration of fluoxetine to rats reduced negative effects of stress on learning in females but not males (47). Further support comes from the finding that the severity of PTSD symptoms was negatively correlated with urinary levels of AVP in men but not women (48). Furthermore, intranasal AVP improved PTSD symptoms in men but not women (48). Determination of possible sex differences in the efficacy of 5-HT– and AVP-active drugs to treat stress-related psychiatric disorders would have an almost immediate clinical impact by guiding drug treatment and development, emphasizing the role of 5-HT–targeted drugs in females and AVP-targeted drugs in males.

Materials and Methods

Animals and Drug Treatment.

Adult male and female Syrian hamsters (Charles River Laboratories Inc. and in-house bred), 8–12 wk old, weighing between 110 and 140 g, were used for all experiments. Hamsters were individually housed in polycarbonate cages (24 × 43 × 20 cm). All hamsters were kept in a 14:10 light/dark cycle with free access to food and water. Nonaggressive intruder (NAI) hamsters were group-housed. All experiments were conducted in accordance with the National Institutes of Health Guidelines for the Use of Animals and were approved by the Georgia State University Institutional Animal Care and Use Committee (49). The following drugs were used in the microinjection experiments: 8-OH-DPAT (78950–78-4; Sigma-Aldrich) and AVP (065–07; Phoenix Pharmaceuticals). Doses of 1 mM and 100 µM of 8-OH-DPAT were used in females and males, respectively. AVP was used at a dose of 0.9 µM in both sexes. The 8-OH-DPAT/AVP mixture was used at 1 mM/0.9 µM doses, respectively, in both sexes. Concentrations were informed by previous studies (13, 22, 50). Drugs were dissolved in sterile physiological saline. Controls were microinjected with 200 nL of saline. Fluoxetine hydrochloride (CAS no. 56296–78-7; Cayman Chemica) dissolved in diH2O was injected intraperitoneally (IP) to females in the following doses: 0 mg/kg (vehicle), 8 mg/kg, and 20 mg/kg. Males received vehicle or 20 mg/kg.

Injection Experiments.

Before surgery, hamsters were housed singly for at least 1 wk. Hamsters were deeply anesthetized with a gaseous solution of 5% (vol/vol) isoflurane mixed with oxygen in an induction chamber and maintained under gaseous anesthesia between 3.00% and 4.00% (vol/vol) isoflurane mixed with oxygen. Hamsters were implanted unilaterally with a 4-mm, 26-gauge guide cannula aimed at the AH using a stereotaxic apparatus. Coordinates were +0.8 mm anterior to bregma, ±1.5 mm from the midline, and –3.5 mm from the top of the skull at an 8° angle (see ref. 38). Hamsters recovered at least 3 d before handling. The next week, hamsters were handled daily, and estrous cycles were monitored in females. The next week, hamsters were tested for aggression. Five minutes before testing, hamsters were microinjected with 200 nL of drug or saline. Injections lasted 30 s, and the needle remained in the guide cannula for an additional 30 s. Hamsters were tested in a neutral arena for 5 min with a smaller NAI of the same sex. Experimental females were tested during diestrus to ensure aggression during testing and NAI females during proestrus to ensure no aggression during testing. After testing, hamsters were euthanized with an overdose of sodium pentobarbital and injected with ink to verify cannula site placement. In the fluoxetine studies, hamsters were housed, handled, and tested as described earlier. Two hours before behavior testing, hamsters were injected IP with fluoxetine or vehicle.

Immunofluorescence.

Hamsters were isolated, handled, and tested in the same way as in the injection experiments. Hamsters were paired with a weight and sex-matched opponent in a resident–intruder paradigm for 15 min, and behavior was analyzed to determine dominance as described in a previous report (46). Controls were moved to an empty, dirty cage belonging to a same-sex hamster. One hour after the start of testing, hamsters were euthanized as described earlier, transcardially perfused, and tissue processed as previously described (51). Brains were cut in 40-µm coronal sections on a cryostat and stored in a cryoprotectant solution (500 mL PBS, 300 g sucrose, 10 g polyvinyl pyrrolidone, 300 mL ethylene glycol) until immunofluorescent processing. Raphe sections were processed using antibodies for 5-HT (20079; Immunostar) and c-Fos (sc-52; Santa Cruz Biotechnology), a marker of neural activation. Hypothalamic sections were processed using antibodies for AVP (T-5048; Peninsula Laboratories) and c-Fos. All immunofluorescent procedures were conducted at room temperature. Sections were washed in PBS four times for 5 min and blocked in 10% normal donkey serum (NDS) with 0.4% of Triton X-100 in PBS for 1 h. Raphe sections were incubated overnight in an antibody solution (ABS; 0.4% of Triton X-100 and 2% NDS in PBS) for 5-HT (1/1,000) and c-Fos (1/1,000). Hypothalamic sections were incubated overnight in ABS for AVP (1/2,500) and c-Fos (1/1,000). Sections were washed in ABS five times for 5 min and incubated in darkness for 2 h in ABS containing secondary antibodies Alexa Fluor 488 and 594 (1/500; Jackson Immunoreserach). All tissue was washed in PBS in darkness four times for 5 min. Tissue was mounted onto Colorfrost Plus Microscope Slides (12-550-17; Fisher Scientific) in PBS, rinsed with diH2O, and coverslipped with Vectashield Hard Set Mounting Medium for Fluorescence (H1400; Vector Laboratories).

Confocal Microscopy and Quantification.

Digital images were acquired with a Zeiss LSM 720 confocal microscope at 8–20× magnification, depending on region size. Using Zeiss ZEN 2012 software, 2-µm interval Z stack images were obtained, and representative images of entire regions from each subject were quantified. Overall adjustments to brightness were applied evenly to channels of all images to maximize clarity. The “Cell Counting” plugin in ImageJ was used for quantification. Cell activation was determined by quantifying cells containing staining for 5-HT or AVP and c-Fos. Digital zooming was used to confirm colocalization. Raphe areas quantified include the DRNa, DRNp, MRN, vrDRNa, and drDRNa. Hypothalamic areas quantified include mSON, NC, and medial PVN. Images in Figs. 4 and 5 are maximum intensity projection images.

Data Analysis.

SPSS v22 was used to analyze all data. Two-way independent analysis of variance (ANOVA), single-variable ANOVA, and independent t tests were all used where appropriate. When appropriate, data were transformed using the square root before commencing with analysis. Graphs are of original data and not transformed data. All post hoc comparisons were a priori. Tests were two tailed, and differences were determined to be significant at P < 0.05.