Make war not love: the neural substrate underlying a state-dependent switch in female social behavior

Summary

Female mice exhibit opposing social behaviors towards males, depending on their reproductive state: virgins display sexual receptivity (lordosis behavior), while lactating mothers attack. How a change in reproductive state produces a qualitative switch in behavioral response to the same conspecific stimulus is unknown. Using single-cell RNAseq, we identify two distinct subtypes of Estrogen receptor-1 (Esr1)-positive neurons in the ventrolateral subdivision of the female ventromedial hypothalamus (VMHvl), and demonstrate that they causally control sexual receptivity and aggressiveness in virgins and lactating mothers, respectively. Between- and within-subject bulk calcium recordings from each subtype reveal that the aggression-specific cells acquire an increased responsiveness to social cues during the transition from virginity to maternity, while the responsiveness of the mating-specific population appears unchanged. These results demonstrate that reproductive state-dependent changes in the relative activity of transcriptomically distinct neural subtypes can underlie categorical switches in behavior associated with physiological state changes.

eTOC:

Internal states can profoundly alter innate behavioral responses to releasing stimuli. Liu et al. identify distinct neural subpopulations within a single hypothalamic nucleus (VMHvl) that causally control female mating vs. aggressive behaviors. Changes in the subpopulations’ relative cue sensitivity underlies a state-dependent switch in social behaviors.

Graphical Abstract

Introduction

Internal states can profoundly alter innate behavioral responses to releasing stimuli, but the underlying cellular and molecular mechanisms remain poorly understood. Female mice show a reproductive state-dependent switch in social behavior towards male conspecifics. Virgins exhibit sexual receptivity (lordosis) towards males (Lindenfors and S.Tullberg, 2011; Hashikawa et al., 2018), while lactating mothers exhibit aggression (and towards females as well) (Gandelman, 1972; Lonstein and Gammie, 2002; Martín-Sánchez et al., 2015). Maternal aggressiveness disappears after pups are weaned and another mating cycle ensues (Bosch, 2013) (Figure 1A). This reversible behavioral switch maximizes mating opportunities for virgins while protecting pups from infanticide. Although hormones such as oxytocin and vasopressin (Lonstein and Gammie, 2002; Bosch, 2013; Oliveira et al., 2021), have been implicated in this behavioral switch, the underlying cellular and circuit mechanisms remain largely unknown.

Figure 1.

Distinct VMH transcriptomic cell types are activated during female mating and aggression.

(A) Female mice adjust social behaviors according to their lactation cycle. The start of the lactating phase is always accompanied by a decrease in sexual behaviors and an onset of aggressive behaviors; at the end of the lactating phase, sexual behaviors resume, and aggression disappears.

(B) Potential cellular mechanisms underlying changes in female VMHvl^Esr1 activity from the virgin to the lactating state. Model 1: The VMHvl^Esr1 neurons controlling socio-sexual behavior are the same in virgin and lactating females, and critical changes occur at their downstream targets to promote mating (in virgins) or aggression (in lactating dams). (In a variant of Model 1 (not shown) different cells control mating vs. aggression, but their state-dependent behavioral output is determined by changes in their downstream targets.) Models 2–4, distinct subsets VMHvl^Esr1 cells control mating in virgins and aggression in lactating dams. Their relative activity differs depending on the animal’s state and determines their behavioral output. The changes in lactating females could involve a decrease in the activity of mating neurons (Model 2); an increase in the activity of aggression neurons (Model 3); or both (Model 4). Models 2–4 can be further subdivided according to the presence (II) or absence (I) of reciprocal inhibitory interactions (likely indirect) between mating and aggression cells. Colored symbols indicate where the lactation-dependent changes occur. The change in the activity of these different subpopulations could reflect cell-intrinsic changes, or changes in input strength.

(C) Schematic of Act-seq protocol.

(D) UMAP plot color-coded by clusters identified in tissue dissected from female ventral VMH (N=19,103; 27 clusters).

(E-G) “Volcano plots” showing Fos expression levels in 27 VMH cell types from the following animals: (E) lactating females exhibiting attacks vs. control, (F) virgin females exhibiting lordosis vs. control, (G) control lactating females vs. control virgin females. Colored dots indicate cell types with Fos expression fold change >2 (x axis cut-off) and P <0.05 (y axis cut-off; gray dashed lines).

(H) Fos expression levels in 12 VMHvl cell types from control lactating females, control virgin females (No intruder), lactating females exhibiting attacks (Aggression) and virgin females exhibiting lordosis (Mating). Colored bars show T-types with statistically significant increases (fold change >2 & P <0.05) in Fos expression relative to no-intruder controls (see also Figure S2C).

(I) Violin plot illustrating expression levels of Npy2r and Esr1 in 12 VMHvl cell types.

(J) Summary of composition of VMHvl cell types. Left, shaded green: VMHvl^{Esr1+,Npy2r−} cells; shaded yellow: VMHvl^Npy2r+ cells; gray: VMHvl^{Esr1−,Npy2r−} cells. Right, green: Percentage of mating-activated cell types (#2, #3), as detected by a significant and >2-fold increase in Fos expression, among VMHvl^{Esr1+,Npy2r−} cells (α); orange: Percentage of aggression-activated cell type (#4) as detected by Fos expression among VMHvl^Npy2r+ cells (β). Fos expression in VMHvl substantially underestimates the fraction of neurons active during a given behavior, as measured by electrophysiological recordings (Lin et al., 2011).

In female rats, the ventrolateral subdivision of the ventromedial hypothalamus (VMHvl) is critical for lordosis behavior (Pfaff and Sakuma, 1979a, 1979b). Within VMHvl, a neuronal subset that expresses the type-1 estrogen receptor (Esr1; also known as ERα) was strongly but indirectly implicated in sexual receptivity (reviewed in (Pfaff, 2017)). More recent studies in mice using genetic tools demonstrated that VMHvl neurons expressing the progesterone receptor (PR) (which co-express Esr1; Kim et al., 2019) are essential both for female mating (Yang et al., 2013; Inoue et al., 2019), as well as maternal aggression (Hashikawa et al., 2017). However while optogenetic activation of these VMHvl^Esr1+/PR+ cells promoted attack in lactating females, it did not promote aggression or lordosis in virgins (Lee et al., 2014; Hashikawa et al., 2017; Inoue et al., 2019).

In both virgins and lactating dams, VMHvl^Esr1+/PR+ bulk calcium activity evoked by conspecifics is equally strong (Hashikawa et al., 2017; Inoue et al., 2019), despite the dramatic difference in the effect of optogenetic stimulation. This has suggested that state-dependent changes in the function of these neurons occurs at their synaptic terminals ( Figure 1B, Model 1; Inoue et al., 2019) or their downstream targets. Alternatively, changes in activity could occur among subsets of VMHvl^Esr1+/PR+ neurons that are obscured in bulk calcium imaging experiments (Figure 1B, Models 2–4).

Indirect evidence has suggested that different subsets of VMHvl^Esr1 neurons may control mating vs. aggression in females (Hashikawa et al., 2017). However, a causal role for these VMHvl subpopulations in sexual receptivity or attack was not demonstrated. Recent single-cell RNA sequencing (scRNAseq) of male VMHvl has revealed that the Esr1⁺/PR⁺ population comprises 6–8 distinct transcriptomic cell types which are differentially activated during specific male social behaviors (Kim et al., 2019). While preliminary studies identified several female-specific VMHvl cell types (Kim et al., 2019), their behavioral role was not investigated.

Here we identify two transcriptomic cell types in female VMHvl that causally control sexual receptivity or aggressiveness, respectively. Within-subject calcium recordings during the transition from virginity to maternity indicate that the aggression-promoting neurons undergo a dramatic increase in their responsiveness to social cues, while the mating neurons do not. Thus, changes in the relative activity of transcriptomically distinct neuronal subtypes underlie a reproductive state-dependent categorical switch in female social behavior.

Results

Distinct VMHvl transcriptomic cell types are activated during female mating and aggression

The lactation state-dependent switch between aggression and mating is a common phenomenon in female mice (Lonstein and Gammie, 2002), but the level of aggressiveness differs depending on genetic background (Hashikawa et al., 2017) or the type of opponent. We used singly housed C57BL/6N female mice, and characterized the level of aggressive behavior in different reproductive states, using a 15-min resident-intruder assay (Figure S1). Females were sequentially tested at three distinct periods in their lactation cycle: Virgin (8–10 weeks); Lactating (“Lac,” 2–5 days after parturition) and Post-lactation (“Post,” 3–6 days after weaning pups), using intruder mice of both sexes (Figure S1A). In the virgin state, none of the tested females attacked either male (0/15) or female (0/21) conspecific intruders. However, almost 80% of lactating resident females displayed attack towards intruder males (11/15) and/or females (16/21). Attack behavior was almost fully abolished in the post-lactation state (Figure S1B, C).

In addition, the majority (15/21) of virgin and post-lactation (17/21) females displayed male-typical mounting and pelvic thrusting behavior towards female intruders (Figure S1D), consistent with a previous report (Hashikawa et al., 2017). This male-typical sexual behavior dramatically decreased in lactating state (2/21) (Figure S1D). Male mice have recently been shown to exhibit two different types of mounting behavior: reproductive (sexual) or dominance (aggressive). These can be distinguished by the presence or absence of ultrasonic vocalizations (USVs), respectively (Karigo et al., 2020). Ultrasonic recordings revealed that the mounting exhibited by virgin and post-lactation females was accompanied by USVs (Figure S1E, F). The meaning of this male-typical behavior in female mice is unclear. Whatever the explanation, C57BL/6N female mice dramatically change their social behaviors and display an on-off switch in aggression regardless of intruder sex, according to their lactation state (Figure 1A).

Previously, we identified multiple VMHvl transcriptomic cell types in males (Kim et al., 2019). Although limited profiling was also performed on female VMHvl, we did not compare females from different reproductive states or attempt to link female cell types to behavior. Here we employed activity-dependent single-cell RNA sequencing (Act-seq, 10x platform) (Wu et al., 2017) in either lactating females exhibiting attack, or in virgins exhibiting lordosis, the female-typical posture of sexual receptivity (Thompson and Edwards, 1971; Pfaff and Sakuma, 1979b) (Figure 1C). Using unsupervised clustering, we identified 27 transcriptomic cell types (T-types) in female ventral VMH (Figure 1D). Clusters #1- #12 are VMHvl cell types, based on the differentially expressed genes (DEGs) Esr1, Npy2r, Dlk1 (for VMHvl) and Nr5a1 (for VMHdm / anterior VMH) (Figure S2A). These data are consistent with those reported in our previous study (Figure S3B), to which we refer the reader for discussion of our statistical analyses and their limitations (Kim et al., 2019).

To identify cell types activated during mating and/or aggression, we examined in our scRNAseq data the expression levels of 139 immediate early genes (IEGs) (Kim et al., 2019) in different T-types within separate cohorts of behaving and control female mice. Among several significantly up-regulated IEGs, Fos exhibited the highest fold-induction (Figure S2C). This analysis identified non-overlapping subsets of aggression- and mating-responding T-types in VMHvl (Figure 1E–H). Clusters #2 and #3 contained cells that were significantly activated in virgins exhibiting lordosis (Figure 1E; p<0.05 compared to home cage controls). Cluster #4 contained cells that were significantly activated in lactating females following attack (Figure 1F; p<0.05, compared to controls). Baseline Fos expression in the absence of an intruder did not differ between states for all VMH cell types (Figure 1G).

Previous work has shown that VMHvl contains several sex-specific T-types (Kim et al., 2019). To determine whether clusters #2, 3 and 4 were female-specific, we computationally integrated our female scRNAseq data with a previous dataset (Kim et al., 2019) that included predominantly male- as well as some female-derived samples (Figure S3). The integration was guided by the larger of the two datasets (Methods). We found that most cells (90.18%) in the female aggression-responding T-type identified here (cluster #4) corresponded to (non-dimorphic) T-type Esr1_4 in the integrated transcriptome, which was also activated during male aggression (Kim et al., 2019). The other male aggression-responding cell type, Esr1_7, was not found in the current female-derived dataset, consistent with our previous observation that it is male-specific (Kim et al., 2019). Therefore, female aggression activates a non-dimorphic cell type that is also activated during male aggression (Esr1_4), while male aggression additionally activates a male-specific T-type (Esr1_7).

In contrast to the aggression-responding cell type, mating-responding cell types #2 and #3 were highly female specific (Figure S3A), comprising a large proportion (11.29%+9.93%) of VMHvl neurons in females (Figure S3B right y-axis). In contrast, cluster #10 (corresponding to cell type Dlk_1) was activated in males but not in females during mating (Kim et al., 2019). Together these data indicated that the female-specific VMHvl cell types identified by Kim et al. (2019) are activated during lordosis, while a different (but non-dimorphic) cell type is activated during male mating.

The foregoing Act-seq data suggested the hypothesis that clusters #2/3 and #4 might function specifically in female mating vs. aggression, respectively. To test this hypothesis, we sought to gain specific genetic access to these cell types. Analysis of differentially expressed genes (DEGs) in these clusters indicated that both expressed Esr1 at varying levels (Figure 1I, lower and S2A), while the NPY receptor Npy2r was expressed in aggression-responding (Figure 1I, upper, orange dot), but not in mating-responding (Figure 1I, lower, green dot) cell types. Overall, among clusters #1-#12 in female VMHvl, 45.7% of the cells expressed Esr1 but not Npy2r (VMHvl^{Esr1+,Npy2r−}). 18.1% of the cells expressed both Esr1 and Npy2r (VMHvl^Npy2r+; Figure 1J), of which 21.1% constitute aggression-activated cell type #4 (Figure S2B).

Based on these data, we designed a genetic intersectional strategy using Npy2^cre (Chang et al., 2015) and Esr1^flp (Karigo et al., 2020) mice to separately label the aggression- and mating-responding subpopulations. We used Esr1^flp and Npy2^cre and Cre-OFF/ Flp-ON (Coff/Fon) AAVs (Fenno et al., 2014, 2020) containing different effector genes to specifically label the mating-responding population in female VMHvl (henceforth referred to as α cells). Most Npy2r-expressing cells are also Esr1⁺, therefore, we used Npy2^cre alone to label the female VMHvl aggression-responding cell type (henceforth referred to as β cells) together with Cre-ON (Con) AAVs. Although Npy2r labels two other VMH T-types -- #5 and #6 – those cell types were not significantly activated above controls during either mating or aggression (Figure 1E, F and S2C). This strategy thus allowed us to independently manipulate α cells or ß cells to investigate their causal roles in mating and aggression.

VMHvl α cells control female receptivity and inhibit maternal aggression.

VMHvl and its constituent PR⁺ neurons (which also express Esr1) are necessary for female mating (Pfaff and Sakuma, 1979a; Yang et al., 2013; Inoue et al., 2019). We therefore first tested whether optogenetic inhibition of VMHvl α (mating-activated) cells would reduce sexual behaviors in virgin females (Figure S4A–E). We expressed eNpHR (Fenno et al., 2020) in α cells of ovary intact virgins, whose estrus cycle we tracked through vaginal cytology, and conducted mating tests when they were in physiological estrus (Figure S4A, B). Optogenetic silencing of α cells significantly reduced the sexual receptivity of estrus virgins during their social interactions with males as measured by lordosis (Figure S4C–E). This result indicates that the normal activity of α cells is necessary for this receptive state.

We next performed gain-of-function perturbations, by optogenetically activating α cells. In previous studies using ovariectomized, estrogen-primed (OVX-E) females, activation of VMHvl PR⁺ or Esr1⁺ neurons did not increase lordosis, perhaps because hormone priming produces a “ceiling effect” on receptivity (Inoue et al., 2019). Therefore, we tested whether selective activation of Esr1 α cells would promote lordosis in sexually naïve ovary intact virgins, which in general displayed a much lower level of sexual receptivity (Thompson and Edwards, 1971; Hardy and DeBold, 1973).

We expressed ChR2 (Fenno et al., 2020) in VMHvl α cells of virgin females, and conducted a mating test when they were physiologically in estrus or diestrus (Figure 2A, B). During estrus, control virgins occasionally (but rarely) performed lordosis. In contrast, during diestrus they strongly rejected male mounting attempts and exhibited escape, kicking and audible vocalizations (Figure 2C–F, Video S1). Strikingly, optogenetic activation of α cells (10–20Hz, 20ms pulse width) effectively enhanced the frequency of lordosis, in both estrus and diestrus virgins (Figure 2C–E, Video S1). The onset of lordosis was not time-locked to photostimulation but was initiated in response to mounting attempts by the male, suggesting that activation of α cells promotes a state of receptivity rather than the motor behavior of lordosis per se. Consistent with this, lordosis was not promoted by α cell activation in solitary animals or during interactions with non-mounting males or females. Thus, activation of α cells significantly enhanced female sexual receptivity during estrus, when females are naturally receptive, and induced it during diestrus (Figure 2C, F and G). These results identify, for the first time, a genetically defined VMHvl cell type whose selective activation can promote or enhance sexual receptivity in both estrus and diestrus virgin females.

Figure 2.

VMHvl α cells control female receptivity and inhibit maternal aggression.

(A) Left, strategy to activate VMHvl^{Esr1+,Npy2r−} cells (α) in virgin females by optogenetics. Right, representative ChR2 expression in VMHvl α cells. Scalebar, 200um.

(B) Behavioral paradigm and illustration of stimulation scheme for one session (also see Methods).

(C) Representative raster plots illustrating light-induced behaviors in tested female and paired male.

(D) Fraction of trials where female exhibited lordosis,

(E) fraction of time female spent in lordosis,

(F) lordosis quotient (lordosis time/male mounting or intromission time),

(G) fraction of time males spent intromitting during light ON or light OFF periods, in estrus or diestrus intact (non-ovariectomized) ChR2/YFP-expressing females.

(H) Strategy to activate VMHvl^{Esr1+,Npy2r−} cells (α) in lactating females by optogenetics.

(I) Representative raster plots illustrating light-induced behaviors in aggressive lactating female.

(J) Average attack probability. In blue, stimulated trials with light ON; in grey, sham trials with light OFF.

(K) Fraction of time lactating female spent attacking during 10-second stimulated or sham periods.

*p<0.05; **p<0.01; ***p<0.001; ****p<0.0001. Mean ± SEM.

Next, we examined the effect of VMHvl α cell activation on aggressive behavior in lactating dams (Figure 2H–K). We activated α cells at the moment of attack onset. Optogenetic activation of α cells rapidly abrogated attack towards female intruders (Figure 2I–K, Video S2). After the abrogation of attack, the stimulated animals continued sniffing the intruder or exhibited social disengagement (e.g., walked away) (Figure 2I, Video S2). Taken together, these data indicate that activation of VMHvl α cells has distinct behavioral effects in virgin vs. lactating females, respectively, facilitating receptivity in the former and inhibiting aggressiveness in the latter.

Activation of VMHvl ß cells evoked female aggression regardless of state

We investigated next the causal role of VMHvl ß (aggression-activated) cells in female aggressive behaviors. We first tested if VMHvl ß cells are necessary for maternal aggression (Figure 3A–E). We silenced ß cells at the onset of attack in lactating females expressing eNpHR (Gradinaru et al., 2010) (Figure 3A, B). Optogenetic inhibition of ß cells immediately interrupted ongoing attack, after which animals exhibited sniffing or social disengagement (Figure 3C–E), similar to the effect of activating α cells. The activity of VMHvl ß cells is therefore essential for maternal aggression.

Figure 3.

Activation of VMHvl ß cells evoked female aggression regardless of state.

(A) Strategy to inhibit VMHvl^Npy2r+ cells (β) in aggressive lactating females by optogenetics.

(B) Representative eNpHR expression in VMHvl β cells. Scalebar, 200um.

(C) Representative raster plots illustrating light-induced behaviors in aggressive lactating.

(D) Average attack probability. In yellow, stimulated trials with light ON; in grey, sham trials with light OFF.

(E) Fraction of time lactating female spent attacking during 10-second stimulated or sham stimulated periods.

(F) Strategy to activate VMHvl^Npy2r+ cells (β) or VMHvl^Esr1+ cells in non-aggressive virgin females by optogenetics.

(G, I) Representative raster plots illustrating light-induced behaviors in virgin female towards (F) female or (H) male intruders.

(H, J) Average attack probability towards (G) female intruder and (I) male intruder. In blue, stimulated trials light ON; in grey, sham trials with light OFF.

(K) Fraction of mice for whom attack was induced by activating β or VMHvl^Esr1+ cells.

(L) Fraction of trials with Npy2r^cre female exhibiting attack.

(M) Fraction of time Npy2r^cre female exhibited attack during a trial.

(N) Fraction of trials with Npy2r^cre female exhibiting USV⁻ mounting.

(O-P) Activation of β cells converted ongoing mating behaviors to attack in virgin females. (O) Behavior raster plot of individual trials. (P) Fraction of females exhibiting lordosis converted to attack.

*p<0.05, **p<0.01, ***p<0.001, ****p<0.0001. Mean ± SEM.

We next tested whether optogenetic activation of ß cells was able to enhance attack in naturally aggressive lactating females (Figure S4F–J). We performed photostimulation (10–20Hz, 5–20ms) when the resident was close to the intruder, but not engaged in attack. Under these conditions, activation of ß cells triggered attack virtually time-locked to the onset of photostimulation (Figure S4G), increasing the fraction of trials in which lactating females exhibited attack towards intruders of either sex (Figure S4H, I).

Previous studies reported that activating VMHvl^Esr1+ neurons in C57BL/6 virgin/non-lactating females was unable to induce attack (Lee et al., 2014; Hashikawa et al., 2017), a result we confirmed (Figure 3K, Esr1⁺). In contrast we found, surprisingly, that activating VMHvl ß cells in virgins was able to trigger fierce time-locked attack (Figure 3G–J), towards both female and male intruders (Figure 3K–M). Activating ß cells also induced male-typical mounting behavior towards intruders of both sexes, in lactating as well as in virgin females (Figure 3N and S4J), as reported previously for activating VMHvl^Esr1+ cells (Lee et al., 2014; Hashikawa et al., 2017). Notably, this optogenetically evoked mounting was not accompanied by ultrasonic vocalizations (USVs), in contrast to naturally occurring female mounting (Figure S1E, F), suggesting that it may be a mild form of aggression or a dominance display (Karigo et al., 2020). Together, these data suggest that activation of VMHvl ß cells does not simply trigger a motor program of attack, but rather induces an aggressive internal state, which is expressed by multiple aggressive behaviors including attack and USV⁻ mounting. This aggressive state can be induced even in naturally non-aggressive virgin females.

Finally, we examined the effect of optogenetically activating VMHvl ß cells on mating in virgin females, during which behavior these cells are normally not strongly activated (Figure 1E, H). Activation of ß cells immediately interrupted ongoing sexual behaviors and efficiently converted them to attack, in estrus virgins (Figure 3O, P, Video S3). Taken together, these data indicate that optogenetic activation of VMHvl ß cells exerts opposite-direction effects in females, inhibiting sexual and promoting aggressive behaviors, mirroring the dual effects of α cell activation, but with an opposite sign. As both α and ß cells are glutamatergic (Kim et al., 2019), their inhibitory effects are likely to be exerted indirectly, through as-yet unidentified GABAergic intermediate cells.

VMHvl α cells, but not ß cells in virgin females selectively respond to male cues

Having established a causal role for VMHvl α and ß cells in female sexual receptivity and aggressivity, respectively, we next investigated the dynamics of their activation during social interactions. Our Act-seq results indicated that these cells were active at some point during female mating or aggressive interactions, respectively (Figure 1E, F). However, due to their lack of temporal resolution it remained unclear during which phase(s) or feature(s) of these social interactions the cells were most strongly active. For example, does their activity mainly reflect pheromonal cues, internal states or specific motor behaviors? To resolve this question, we recorded optically bulk calcium signals (hereafter referred to as “fiber photometry” (Lerner et al., 2015)) from the VMHvl α or ß cells to examine continuously their activity dynamics during social interactions.

Initially, we investigated which stimuli activate the VMHvl α cells most strongly. First, we expressed calcium sensor GCaMP6m (Fenno et al., 2020) in virgin females, and performed fiber photometry recordings from VMHvl α cells during the appetitive phase of social interactions (before the initiation of consummatory mating) (Figure 4A–C). Within a given session, we introduced a female, male or toy mouse into the female’s home cage (in a randomly shuffled order), allowed them to interact freely for 3 minutes, and compared the neural responses to the different intruders. The initial encounter, defined as the first close sniffing interaction between the female resident and the intruder (Karigo et al., 2020). VMHvl α cells showed a significantly higher response during an initial encounter with a male than with either a female, or a toy mouse (Figure 4E, initial encounter).

Figure 4.

VMHvl α cells, but not ß cells in virgin females selectively respond to male cues.

A) Schematic illustrating fiber photometry recording in VMHvl.

(B) Schematic illustrating behavioral test of social interaction in virgin females.

(C-D) Representative GCaMP6m expression in (C) VMHvl α and (D) β cells. Scalebar, 200um.

(E-F) Left, Peri-event time histogram (PETH) (E) α cells or (F) β cells, aligned to initial encounter with intruders. Initial encounter occurred immediately after intruder introduction. Right, area under the curve (AUC) represents timespan 0 to 30 seconds after the initial encounter.

(G) Male Preference Index (MPI) of initial encounter from α and β cells. MPI = (AUC of initial encounter with male - AUC of initial encounter with female) / (AUC of initial encounter with male + AUC of initial encounter with female).

(H-I) Left, PETH from (H) α cells or (I) β cells, aligned to sniffing intruder; these are defined as sniffing bouts occurring > 1 minute after intruder introduction. Sniffing bouts followed by male mounting were excluded. Right, AUC represents timespan 0 to 3 seconds after sniffing intruder.

(J) MPI of sniffing intruder from α and β cells. MPI = (AUC sniffing male - AUC sniffing female) / (AUC sniffing male + AUC sniffing female).

(K-L) Left, PETH from (K) α and (L) β cells, aligned to first sniff of urine. Right, Peak PETH between time 0 to 10 seconds relative to first sniff of urine.

(M) MPI of first sniff of urine from α and β cells. MPI = (Peak of sniffing of male urine − Peak of sniffing of female urine) / (Peak of sniffing of male urine + Peak of sniffing of female urine).

**p<0.01; ***p<0.001; ****p<0.0001; ns, not significant. Mean ± SEM.

We also measured the responses of α cells during subsequent sniffing of the intruder following the initial encounter (all sniffing bouts that were followed immediately by consummatory male mounting were excluded). We observed elevated activity during almost all bouts of sniffing males, but not during sniffing of females or the toy (Figure 4H, sniff intruder). However, the magnitude of the response during these subsequent sniff bouts was typically lower than that during the initial encounter (Karigo et al., 2020). To confirm that the male-preferred response is caused by olfactory cues, we compared α cell responses when the female was presented sequentially with samples of female urine, male urine or water (order of stimuli randomly shuffled). The α cells showed a high response to male urine, but low responses to female urine or water, during the initial sniff (Figure 4K). These results indicate that VMHvl α cells are preferentially activated by male cues, even in the absence of mating behavior.

To investigate if VMHvl ß cells also display a sex-preferred response, we performed the same fiber photometric recordings in ß cells in virgins (Figure 4D). In contrast to α cells, the responses of ß cells during the initial encounter with a male were only slightly and not significantly higher than the initial responses to females (Figure 4F). Moreover, the activity of ß cells was not significantly different during subsequent bouts of sniffing a male or female (Figure 4I). Accordingly, ß cells showed similar response to male or female urine (Figure 4L). These responses overall appeared lower than those of α cells exposed to males or male urine (cf. Figure 4E, H, K vs. 4F, I, L).

To quantitatively compare the responses of VMHvl α and ß cells to males vs. females, we calculated a “male-preference-index” (MPI) (Figure 4G, J and M). α cells exhibited a high MPI during the initial encounter, sniff intruder and sniff urine tests. By contrast, ß cells did not show any significant sex preference in any of the comparisons (Figure 4G, J and M). Thus, the preferential response of VMHvl^Esr1+/PR+ neurons to males reported previously (Inoue et al., 2019) is due almost entirely to the activity of the α cell subpopulation.

VMHvl α cells are highly active in virgin females during mating behaviors

We next investigated if VMHvl α cells are also active during consummatory sexual behaviors in a mating interaction. During the recording session, we introduced a sexually active adult male to the female home cage and allowed the two mice to interact freely for up to 30 mins (Figure 5A). The activity of α cells dramatically increased when the female was mounted by the male and displayed lordosis behavior (Figure 5B, C). The high level of activity was sustained until the mounting bouts were finished. Notably, the responses of α cells to male mounting or intromission (consummatory phase) were significantly higher than the responses to sniffing male (appetitive phase) (Figure 5E). These results indicate that α cells are not only activated by male olfactory cues, but are even more strongly activated during the consummatory phase of mating, consistent with the results of our optogenetic stimulation experiments.

Figure 5.

VMHvl α cells are highly active in virgin females during mating behaviors.

(A) Schematic illustrating fiber photometry recording in VMHvl and behavioral test in virgin females.

(B-C) PETH of GCaMP6m fluorescence in α cells aligned to onset of (B) male mounting and (C) lordosis.

(F-G) PETH in β aligned to onset of (F) male mounting and (G) lordosis.

(D, H) Representative normalized calcium traces from (D) α cells and (H) β cells during mating interaction. Colored shading marks behavioral episodes.

(E, I) Peak of PETH from (E) α cells and (I) β cells aligned to onset of appetitive phase (sniff) or consummatory phase (male mounting or intromission) during mating interaction.

*p <0.05; ns, not significant. Mean ± SEM.

In contrast, ß cells were not active during female mating behaviors, consistent with our Act-seq results. Compared with the α cells, ß cells showed very low responses at both the onset of mounting by the male or during lordosis (Figure 5F–I).

Taken together, these data indicate that in virgin females, VMHvl α cells preferentially respond to male chemosensory cues, and are activated during both the appetitive (sniffing) and consummatory (mounting or intromission) phases of female mating interaction. Conversely, ß cells display similarly weak responses to both male and female cues, and are not active during sexual behaviors.

VMHvl ß cells are highly active in lactating females during aggressive interactions

To examine how VMHvl ß cells respond during female aggressive interaction, we recorded their neural activity using fiber photometry in lactating females (Figure 6A). We observed that the activity of ß cells was high at the onset of attack or sniffing of either the female or the male intruder (Figure 6B–F). The responses to female and male intruders are both high, consistent with the similar frequency of attack towards both types of intruders (Figure S1B, C). Around 20% of lactating females did not become aggressive (Figure S1B, C). In those non-aggressive females, the activity of ß cells at the onset of sniffing was significantly lower than the activity in aggressive females (Figure 6G, H). Moreover, even in aggressive females, the activity was significantly higher at the onset of sniffing bouts which were followed by attack, than during sniffing bouts which were not followed by attack (sniffing only) (Figure S5B, C) (Falkner et al., 2014).

Figure 6.

VMHvl β cells are highly active in lactating females during aggressive interactions.

(A) Schematic illustrating fiber photometry recording in VMHvl β cells and α cells, and behavioral test of aggressive behaviors in lactating females.

(B) Representative normalized GCaMP6m trace from β cells during aggressive interaction with female intruder. Colored shading marks behavioral episodes.

(C-F) PETH of β cell activity aligned to onset of (C) attack and (D) sniffing of female intruder, or (E) attack and (F) sniffing of male intruder. Sniffing includes all sniffs, whether or not followed by attack.

(G-H) β cell responses during sniffing of (G) female or (H) male intruders in aggressive (colored lines) vs. constitutively non-aggressive (gray lines) lactating females. Left, PETH of β cell activity aligned to onset of sniffing. Data from aggressive females is replicated from panels (D) and (F) to facilitate comparisons. Right, quantification of PETH peak and area under the curve (AUC).

(I) Representative normalized GCaMP6m trace from α cells during aggressive interaction with female intruder.

(J-M) PETH of α cell activity aligned to onset of (J) attack and (K) sniffing of female intruder, or (L) attack and (M) sniffing of male intruder.

*p<0.05; **p<0.01; ns, not significant. Mean ± SEM.

We also examined whether there is a correlation between IEG expression in ß cells and aggressiveness using act-seq (Figure S6H). We performed transcriptomic and IEG analysis in females that did or did not attack during social interactions. We found that the aggression-responding cell type #4 was not significantly activated in non-aggressive females (Figure S5I–K). Together, our results indicate that female VMHvl ß cells are active during maternal aggression, regardless of intruder sex, that that their level of activity correlates with the animals level of aggressiveness.

We also examined the activity of α cells during aggressive interactions (Figure 6A). α cells showed relatively low activity at onset of attack or sniffing of either a female or male intruder (Figure 6I–M). Moreover, there was no significant difference in activity during sniffing female or male intruders whether or not it was followed by attack (unlike the case for ß cells; Figure S5D, E). A direct comparison of the activity of α cells during aggressive vs. mating behaviors towards the same social target (Figure S5F, attack male vs. mounted by male), revealed that they were significantly more active during mating behavior. Conversely, the opposite was observed for the ß cells (Figure S5F, G).

Taken together with the results of our functional perturbation experiments, these data demonstrate that female VMHvl α cells and ß cells both causally control, and are specifically active during, mating and aggressive behaviors, respectively.

Neural response of VMHvl ß cells to social cues is lactation state dependent

The identification of distinct subsets of VMHvl neurons that specifically control female mating and aggressive behaviors raised the question of whether one or both of these populations exhibits changes in activity during the transition from the virgin to the lactating state. In principle, the switch from sexual receptivity to aggression could involve a decrease in α cell activation, an increase in ß cell activation, or a combination of both (Figure 1B). To address this question, we performed longitudinal, within-subject fiber photometric analysis of bulk calcium activity in either VMHvl α cells or ß cells in initially virgin females, and followed their activity into the lactating and post-lactation states over a 9 to 10-week period (Figure 7 and Methods).

Figure 7.

Neural response of VMHvl ß cells to social cues is lactation state dependent.

(A) Schematic illustrating fiber photometry recording in VMHvl β cells.

(B) Timeline of longitudinal behavioral tests and recordings in the same individuals across their lactation cycle. Each female was tested 3 times in the virgin, lactating (Lac) and post-lactating (Post) phases. Parturition date was counted as p0.

(C, E) Representative examples of normalized GCaMP6m traces from the same individual female resident across her lactation cycle, during free interactions with (C) male or (E) female intruders.

(D, F) Left, PETHs from three phases of lactation cycle aligned to onset of sniffing of (D) male or (F) female. Right, quantification of PETH peak and AUC. No change of statistical significances after removing the highest data points in each group (see Supplemental Table 1).

(G, J) Representative of normalized GCaMP6m traces from the same individual female resident across lactation cycle during no-contact interaction with (G) male or (J) female intruder protected by an inverted pencil cup (schematic).

(H, K) AUC of neural activity recorded from three phases of lactation cycle during the first 60 seconds relative to introduction of pencil cup-protected (H) male or (K) female.

(I, L) Averaged AUC or peak of neural activity during each 10-second time bin relative to introduction of pencil cup-protected (I) male or (L) female, from the three phases of the lactation cycle.

(M) Left, Model 3 II, related to Figure 1B. Right, schematic illustrating the information flow during social interaction in virgin and lactating females: from sensory inputs to brain circuits to emotional states to behavioral output. Line thickness represents information strength. Blue: male intruder; Pink: female intruder.

**p<0.01, ***p<0.001; Mean ± SEM.

We expressed GCaMP6m in VMHvl α or ß cells of virgin females (Figure 7A and S6A). Then we performed recording of these two cell populations during three successive epochs across their lactation cycle: virgin, lactating (lac) and post-lactation (post), and compared the level of bulk calcium activity across states within the same individuals (Figure 7B and S6B). In virgin state, GCaMP6m signals in ß cells were relatively lower during exposure to either intruder males or females, but increased significantly in lactating state (Figure 7C–F and S7A–B). This increase in signals was not due to an increase in baseline GCaMP6m expression, because it returned to levels seen in virgins when the females entered post-lactation state (Figure 7C–F and S7A–B). In contrast, bulk calcium signals in VMHvl α cells during interactions with male or female intruders did not change between the virgin and lactating states (Figure S6C, D, F and G). However, the MPI of α cells for sniffing intruders was significantly lower in lactating mothers than in virgins (Figure S6H, left).

Because lactating females display a different behavioral repertoire, compared with virgins or post-lactation females, it was unclear whether the observed change in ß cell activity simply reflected this switch in behavior, or a modification of their responses to sensory cues. To discriminate those two possibilities, we used a pencil cup-protected intruder to create a no-contact interaction situation (Figure 7G–L and S7C–F). In this assay, tested females could only sniff the intruder but could not display any consummatory behaviors such as attack or lordosis.

In the no-contact interaction, ß cells still displayed a significantly higher response to either male or female intruders in lactating dams, compared to virgins or post-lactational mothers (Figure 7G–L). The sniffing pattern (Figure S7C, E) and total sniffing time (Figure S7D, F) during each session was not different in all three tested states, indicating that the higher neural activity observed in the lactating state is not simply due to a longer interaction. We also compared the neural responses to urine in females in different lactation states (Figure S7G–J). Consistent with our results using intact intruders, ß cells displayed a significantly higher response to either female or male urine in the lactating state, compared to their responses in the other states (Figure S7G–J). These results support the idea that female VMHvl ß cells, but not α cells, display a state-dependent, reversible change in responsivity to social cues that correlates with the animal’s change in behavioral response to intruders.

Discussion

Transcriptomic cell types have emerged as fundamental units of brain organization and function (Zeng and Sanes, 2017; Li, Tyler and Haydar, 2020), but whether and how specific cell types contribute to behavioral plasticity remains unclear. We have investigated the neural control of a reproductive state-dependent switch in female social behaviors (Yang et al., 2013; Hashikawa et al., 2017; Inoue et al., 2019). Our data demonstrate that this behavioral switch reflects a reversible change in the functional properties of distinct transcriptomic cell types. These findings reveal a role for cell type-specific regulation in the state-dependent control of social behavior, and open the way to elucidating the underlying molecular- and circuit-level mechanisms.

Identification of genetically distinct VMHvl^Esr1 subpopulations causally controlling female mating vs. aggression

The Esr1⁺/PR⁺ population in VMHvl has previously been shown to be necessary for mating and aggression in both male and female mice (Yang et al., 2013; Lee et al., 2014; Hashikawa et al., 2017; Karigo et al., 2020). Indirectly data have suggested that these behaviors are controlled by different cell types (Hashikawa et al., 2017). Here we assign causal roles in mating or aggression to two transcriptomically distinct Esr1⁺ subtypes in female VMHvl. We demonstrate that these cell types are not only essential for these behaviors, but that their optogenetic activation can promote them -- even under conditions where the behaviors are not naturally expressed. For example, ß cell activation induces aggressiveness in C57BL/6 virgin females, which normally never exhibit this behavior. Indeed, prior attempts to evoke aggression in virgins by stimulation of the bulk Esr1⁺ population were unsuccessful (Lee et al., 2014; Hashikawa et al., 2017). This led some to conclude that state-dependent constraints on aggression in virgins are dominant over Esr1⁺ neuronal function (Hashikawa et al., 2017). Our results show that, to the contrary, aggression can be evoked in virgins, if a specific subtype of Esr1⁺ neurons is activated. We speculate that the prior failure to activate aggression by stimulation of the entire Esr1⁺ population may reflect antagonistic interactions between α and ß cells that cancel each other out.

Similarly, prior studies showed that activation of Esr1⁺ or PR⁺ VMHvl neurons failed to enhance sexual receptivity in virgins (Lee et al., 2014; Hashikawa et al., 2017; Inoue et al., 2019). In contrast, we show here that activation of α cells can promote receptivity in estrus virgins, and induce it in normally unreceptive diestrus virgins. Several factors may explain this discrepancy. First, our genetic intersectional strategy excluded the Npy2r⁺ (ß cell) subset, which inhibited mating when stimulated on its own. Activation of ß cells may therefore nullify the ability of α cells to promote receptivity, when both populations are co-stimulated in the bulk Esr1⁺ population. Second, we used intact virgins rather than OVX-E females, eliminating potential “ceiling effects” on receptivity that could obscure the effect of α cell stimulation (Inoue et al., 2019). Our results indicate that even in diestrus females, α cells can activate downstream targets that control sexual receptivity (although quantitative modulation of the efficiency of this activation likely contributes as well (Inoue et al., 2019)).

Interestingly, while activation of α cells could override the lack of sexual receptivity during diestrus, it was not able to do so in lactating females exposed to male intruders (Figure S4K). This was not due to an intrinsic inability to activate α cells in mothers, because their stimulation inhibited aggression. Several possibilities may explain the different behavioral phenotypes of α cell activation in virgins vs. lactating females. One is that the high level of endogenous ß cell activity in dams might antagonize the ability of α cell activation to promote receptivity. Another is that the intrinsic cellular properties of α cells or their downstream targets might be different in lactating females vs. virgins. Further studies will be required to distinguish these alternatives.

State-dependent changes in the stimulus-responsivity of female VMHvl aggression neurons

If optogenetic activation of ß cells can evoke aggression in virgin females, why are virgins not naturally aggressive? Our calcium recordings indicate that in virgins, ß cells are much less responsive to male cues than are α cells. Longitudinal within-subject recordings revealed that ß cells dramatically increase their responses to males during the transition from virginity to maternity. Importantly, this increase reverses after the females become post-lactational, indicating that it is not simply due to accumulated GCaMP expression. In contrast, α cells do not exhibit such a dramatic change. These data support Model 3, among the 4 classes of cellular models that could in principle explain this behavioral switch (Figure 1B and 7M). The ability of optogenetic ß cell activation to suppress sexual receptivity in virgins further suggests that these cells may normally inhibit sexual receptivity in lactating females, where they are more active (Figure 7M).

Our observations raise the question of the neural and molecular mechanisms by which ß cells become more responsive to male cues in lactating females. In principle, the excitability of these neurons could increase, due to a loss of local inhibitory input or to changes in the intrinsic biophysical properties of these cells. Alternatively, there could be a state-dependent gating event “upstream” of ß cells, which increases the amount of excitatory input they receive from male-responsive neurons elsewhere in the circuit. Answering this question is an important subject for future investigation. The identification of hormonal or other factors that trigger these changes during pregnancy and/or parturition will be of interest as well.

In males, VMHvl^Esr1 neuronal activity reflects both intruder sex and social behavior (Remedios et al., 2017). It has been difficult to separate functionally the encoding of these two features, because intruder sex and social behavior are tightly coupled in males (Lee et al., 2014; Remedios et al., 2017; Karigo et al., 2020). In lactating females, by contrast, aggressiveness can be triggered by either male or female intruders. The fact that ß cells are comparably activated by males and females in lactating dams, but are more strongly activated in aggressive than in non-aggressive mothers (Figure 6 and S5), strongly suggests that these cells encode aggressiveness rather than simply intruder sex (although we cannot exclude the existence of sex-specific subsets of ß cells). By extension, the same may be true in males.

Sex specific cell types and functions

VMHvl is highly sexually dimorphic, with respect to gene expression and connectivity (Xu et al., 2011; Yang et al., 2013; Knoedler and Shah, 2018). More recently, single-cell RNAseq has revealed that subsets of VMHvl^Esr1 neurons are male- or female-specific (Kim et al., 2019). Here we assign a functional role to a female-specific cell type (Tsix_Esr1_2) in sexual receptivity. This is concordant with the fact that mating behavior is highly sexually dimorphic – lordosis is only performed by females and not by males.

In males, aggression activates both a dimorphic VMHvl^Esr1 cell type (Esr1_7) and a non-dimorphic cell type (Esr1_4) (Kim et al., 2019). In females, by contrast, only the non-dimorphic cell type is active during aggression. Similarly, in Drosophila there is evidence for a non-dimorphic aggression-promoting cell type, CAP, that may control an aggressive motive state (Chiu et al., 2021). However, there are also dimorphic cell types that promote aggression in each sex. In females, the aggression-specific cell type pC1d (Deutsch et al., 2020; Schretter et al., 2020, Chiu et al., 2021) is located in a cluster that also contains a closely related subtype, pC1a, which controls female sexual receptivity (Wang et al., 2021). Thus, in both female mice and female flies, there is a close anatomic and genetic relationship between the neuronal subtypes that control sexual receptivity or aggression.

STAR METHODS

RESOURCE AVAILABILITY

Lead contact

Further information and requests for reagents and resources should be directed to and will be fulfilled by the Lead Contact, David J. Anderson (wuwei@caltech.edu).

Materials availability

This study did not generate new unique reagents.

Data and code availability

• Single-cell RNA-seq data have been deposited at Mendeley (DOI: 10.17632/w25zm9cn6x.1) and are publicly available as of the date of publication. Neural manipulation and recording data reported in this paper is available from the lead contact upon request.

• This paper does not report original code.

• Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

EXPERIMENTAL MODEL AND SUBJECT DETAILS

Mice

All experimental procedures involving the use of live mice or their tissues were carried out in accordance with NIH guidelines and approved by the Institute Animal Care and Use Committee (IACUC) and the Institute Biosafety Committee (IBC) at the California Institute of Technology (Caltech). All mice in this study, including wild-type and transgenic mice, were bred at Caltech. Single-housed C57BL/6N female mice (2–5 months) were used as experimental mice. Group-housed sexually naïve BALB/c male and female mice (6~8 weeks) were used as intruder mice for resident-intruder test (see resident-intruder test section below), unless specifically indicated in the text. Single-housed, sexually experienced C57BL/6N male mice were used for the mating test (see mating test section below). Npy2r^cre mice (Jackson Laboratory stock no. 029285) (=N1), Esr1^cre mice, Esr1^flp mice (>N10) were backcrossed into the C57BL/6N background and bred at Caltech. Heterozygous Npy2r^cre, Esr1^cre or double heterozygote Esr1^flp/+Npy2r^cre/+ mice were used for cell-specific targeting experiments and were genotyped by PCR analysis using genomic DNA from tail tissue. All mice were housed in ventilated micro-isolator cages in a temperature-controlled environment (median temperature 23°C, humidity 60%), under a reversed 11-h dark–13-h light cycle, with ad libitum access to food and water. Mouse cages were changed weekly. For the animals used for longitudinal tests across the lactation cycle, virgin females aged 8~10 weeks were singly housed for at least 3 days before conducting behavioral tests or mating. Once the pregnancy was visible, the male mouse was removed from the female’s home cage. The first day pups were found was counted as postpartum day 0 (p0). Behavioral tests in the lactating state were conducted on p2 to p5. Pups were euthanized after p7. The natural lactation period for mice is 21 days; behavior tests in the post-lactation state were therefore conducted on p24 to p27.

Viruses

The following AAVs were used in this study, with injection titers as indicated. Viruses with a high original titer were diluted with clean PBS on the day of use. AAV2-EF1a-DIO-hChR2(H134R)-EYFP (4 × e12 genome copies per ml), AAV5-EF1a-DIO-eNpHR3-EYFP (4.5 × e12), AAV5-EF1a-DIO-YFP (6.5 × e12), AAV-DJ-hSyn-Coff/Fon-hChR2(H134R)-EYFP (4 × e12), and AAV-DJ-Syn-Coff/Fon-EYFP (3.4 × e12) were purchased from the UNC vector core. AAV1-Syn-Flex-GCaMP6m (2.1 × e13) and AAV8-nEF-Coff/Fon-eNpHR3.3-EYFP (5.6 × e12) were purchased from Addgene. AAV-DJ-EF1a-Coff/Fon-GCaMP6m (4.5 × e12, Addgene plasmid) was packaged at the HHMI Janelia Research Campus virus facility. “Con” indicates Cre-ON virus; “Coff/Fon” indicates Cre-OFF/FLP-ON virus.

METHOD DETAILS

Behavior tests

All behavioral experiments were performed in the animals’ housing cages under red lighting during the animals’ subjective night phase, using the previously described behavior recording setup (Hong et al., 2015). Both top and front views of the behavior were filmed at 30 Hz using video recording software, StreamPix7 (Norpix). Audio recordings were collected at a 300-kHz sampling rate using an Avisoft-UltraSoundGate 116H kit with a condenser ultrasound CM16/CMPA microphone (Avisoft-Bioacoustics), positioned 45 cm above the arena. The audio and video recordings were synchronized to begin at the same time via a signal generated by StreamPix7. Mouse cages were not cleaned for a minimum of one day before the behavioral test, to retain the odors of the resident mouse. Behavior tests were performed during the subjective dark period of the mouse housing room day–night cycle.

Resident-intruder test

Sexally naïve group-housed Balb/c mice were used as intruders. On the test day, the tested female was introduced to the behavior recording setup in her home cage and allowed to rest for at least five minutes. For lactating females, pups were removed from cage before starting acclimation. Then a freely moving or pencil cup-confined male or female or was introduced to the tested female cage. The animals were allowed to interact freely for 3–30 minutes, as described in the text. An experimenter interrupted the interaction between animals if excessive aggression or male ejaculation were observed.

Mating test

For optogenetic experiments investigating α cell function in mating (Figure 2A–G and S4A–E and K), to acquire the maximum sexual activity in male during the limited time window for behavior test, single-housed and sexually experienced C57BL/6N male mice were used as resident and sexually primed before test. On the test day, a male was introduced to the behavior recording setup in his home cage and allowed to rest for at least five minutes. A random female was introduced to male cage and allowed to freely interact with the male mouse. When three male mounting attempts were observed, the random female was removed from male cage. Then the sexually primed male and the tested female were allowed to rest in the testing room for at least five minutes before starting behavior tests. The tested female was then introduced to the male cage. The animals were allowed to interact freely for up to 30 minutes. An experimenter interrupted the interaction between the animals if ejaculation was observed.

Urine presentation

For urine presentation experiments (Figure 4K–M and S7G–J), subject female mice were moved to a new cage one day before the experiment. Group-housed Balb/c female or male mice were used as urine donors. Immediately before testing, the donor mouse was lifted by the cervical region and the genital area was gently wiped with the top part of a Q-tip to absorb urine seeping from the anogenital region. The urine-soaked Q-tip was placed in the cage for two minutes, far away from the current location of the resident female, and then removed from the cage. Female and male urine samples were shuffled and presented to each female subject mouse two to three times, with at least a two-minute interval between presentations. Urine from different donors was used every time.

Single-cell isolation and library construction

On the test day, female subjects were conducted a 15-minute resident-intruder test. A group-housed Balb/c female were used as intruders to induce attack in subject lactating females, while a single-housed, sexually primed Balb/c male was used as resident to induce lordosis in subject virgin females. After the social interactions, the Balb/c mice were taken out of the cage, and the female subjects were left in the cage for 35–55 minutes. Lactating females exhibiting attack, virgin females exhibiting lordosis during the interaction, or control (no intruders) females were then collected immediately. We isolated single cells from the mouse brain as previously described (Wu et al., 2017), with some modifications. Subject females (4 brains collected at a time) were anaesthetized with isoflurane and transcardially perfused with cold NMDG-ACSF (adjusted to pH 7.3–7.4) containing CaCl2 (0.5 mM), glucose (25 mM), HCl (92 mM), HEPES (20 mM), KCl (2.5 mM), kynurenic acid (1 mM), MgSO4 (10 mM), NaHCO3 (30 mM), NaH2PO4 (1.2 mM), NMDG (92 mM), sodium L-ascorbate (5 mM), sodium pyruvate (3 mM), and thiourea (2 mM), bubbled with carbogen gas. The brain was sectioned at 350 mm using vibratome (VT1000S, Leica Microsystems) on ice, and the regions of interest were microdissected under a dissecting microscope from two consecutive sections (−1.22 to −1.94 from Bregma (Franklin and Paxinos, 2008)). The microdissected tissues were collected in a microcentrifuge tube containing NMDG-ACSF with 30 mM actinomycin D on ice to suppress further immediate early gene (IEGs) activation (Wu et al., 2017). Thereafter, the sections were transferred to a new microcentrifuge tube for papain digestion (60 U/mL, Sigma, P3125–250MG; pre-activated at 34°C for 30 min) in Trehalose-HEPES-ACSF (adjusted to pH 7.3–7.4) containing actinomycin D (15mM), CaCl2 (2 mM), L-cysteine (2.5 mM), EDTA (0.5mM), glucose (25 mM), HEPES (20 mM), KCl (2.5 mM), kynurenic acid (1 mM), MgSO4 (2 mM), NaCl (92 mM), NaHCO3 (30 mM), NaH2PO4 (1.2 mM), trehalose (2.5% w/v). During incubation for enzymatic digestion (30 minutes at room temperature), the solution was gently mixed a few times every 5–10 minutes. After incubation, this solution was replaced with cold Trehalose-HEPES-ACSF containing Egg White/BSA ovomucoid inhibitor (3 mg/ mL, Worthington, OI-BSA) and DNase I (25 U/mL, Thermo scientific, 90083). The tissue pieces were dissociated into single cells by gentle, successive trituration through 300-mm and 150-mm Pasteur pipettes with polished tip openings. After trituration and filtering through a 40 mm cell strainer, single cells were pelleted at 300 g for 5 minutes at 4°C, and the supernatant was carefully removed; the cells were resuspended with cold Trehalose-HEPES-ACSF and filtered through a 20 mm filter. Cells were pelleted again at 300 g for 5 minutes at 4°C and resuspended with Resuspension-HEPES-ACSF containing BSA (0.05%), CaCl2 (2 mM), glucose (25 mM), HEPES (20 mM), KCl (2.5 mM), kynurenic acid (1 mM), MgSO4 (1 mM), NaCl (117 mM), NaHCO3 (30 mM), NaH2PO4 (1.2 mM) (osmolarity verified to be within 10 mOsm of Trehalose-HEPES-ACSF). After manually determining the cell concentration using a hemocytometer, suspensions were further diluted to desired concentrations (600–1,000 cells/mL) if necessary. The appropriate volume of reverse transcription (RT) mix was added to target 10,000 cells recovered and loaded into the chip. Chromium Next GEM Single Cell 3ʹ GEM, Library & Gel Bead Kit v3.1 (PN-1000121), Chromium Next GEM Chip G Single Cell Kit (PN-1000120), and Chromium i7 Multiplex Kit (PN-120262) were used for all downstream RT, cDNA amplification (11 PCR cycles), and library preparation as instructed by the manufacturer (Chromium Next GEM Single Cell 3ʹ Reagent Kits v3.1 User Guide).

10x seq data processing and quality control

Libraries were sequenced on an Illumina HiSeq4000 or NovaSeq6000 (paired-end with read lengths of 150 bp) and Illumina sequencing reads were aligned to the mouse pre-mRNA reference transcriptome (mm10) using the 10x Genomics CellRanger pipeline (version 3.0.2) with default parameters. Cells that met any one of the following criteria were filtered out for downstream processing in each 10x run: less than 600 detected genes (for UMI count > 0), UMI counts exceeding 50,000 (potential multiplets), or proportion of the UMI count attributable to mitochondrial genes exceeding 15%. Doublets of different cell classes were further removed by first classifying cells into broad cell classes (neuronal versus non-neuronal) based on the co-expression of any pair of their marker genes (Stmn2 for neurons; Cldn5 for endothelial cells; C1qc for microglia; Opalin for oligodendrocytes; Gja1 for astrocytes; Pdgfra for OPCs; Mustn1 for mural cells). VMH cells are selected based on marker genes Slc17a6, Adcyap1, Fezf1, Dlk1 and Nr5a1, as reported in our previous study (Kim et al., 2019).

Stereotaxic surgery

Surgeries were performed on adult Npy2r^cre, Esr1^cre or Esr1^flp/+Npy2r^cre/+ mice aged 8–12 weeks. Mice were anaesthetized with isoflurane (0.8–5%) and placed on a stereotaxic frame (David Kopf Instruments). Virus was then injected into the target area using a pulled glass capillary (World Precision Instruments) and a pressure injector (Micro4 controller, World Precision Instruments), at a flow rate of 20 nl/min. The injection volumes were 100–300 nl. The Stereotaxic injection coordinates were based on the Paxinos and Franklin atlas (VMHvl, anterior–posterior: −4.68, medial–lateral: ±0.72, dorsal–ventral: −5.7). For optogenetic and fiber photometry experiments, single or dual optic fibers (optogenetics: diameter 200 μm, N.A., 0.22; fiber photometry: diameter 400 μm, N.A., 0.48; Doric lenses) were then placed above the virus injection sites (optogenetics: 500 μm above; fiber photometry: 150 μm above) and fixed on the skull with dental cement (Parkell). Mice were singly housed after surgery and were allowed to recover for at least 3 weeks before behavioral testing.

Fiber photometry

The fiber photometry setup was as previously described in earlier research (Lerner et al., 2015) with minor modifications. We used 470 nm LEDs (M470F3, Thorlabs, filtered with 470–10 nm bandpass filters FB470–10, Thorlabs) for fluorophore excitation, and 405 nm LEDs for isosbestic excitation (M405FP1, Thorlabs, filtered with 410–10 nm bandpass filters FB410–10, Thorlabs). LEDs were modulated at 208 Hz (470 nm) and 333 Hz (405 nm) and controlled by a real-time processor (RZ5P, Tucker David Technologies) via an LED driver (DC4104, Thorlabs). The emission signal from the 470 nm excitation was normalized to the emission signal from the isosbestic excitation (405 nm), to control for motion artefacts, photobleaching and levels of GCaMP6m expression. LEDs were coupled to a 425 nm longpass dichroic mirror (Thorlabs, DMLP425R) via fiber optic patch cables (diameter 400 μm, N.A., 0.48; Doric lenses). Emitted light was collected via the patch cable, coupled to a 490 nm longpass dichroic mirror (DMLP490R, Thorlabs), filtered (FF01–542/27–25, Sem- rock), collimated through a focusing lens (F671SMA-405, Thorlabs) and detected by the photodetectors (Model 2151, Newport). Recordings were acquired using Synapse software (Tucker Davis Technologies).

On the test day, after at least 5 minutes of acclimation under the recording setup, the female was first recorded for 5 minutes to establish a baseline. Then behavior assays were proceeded and fluorescence were recorded for the indicated period of time, as described in the text.

All data analyses were performed in Python. Behavioral video files and fiber photometry data were time-locked. F_n was calculated using normalized (405 nm) fluorescence signals from 470 nm excitation. F_n(t) = 100 × [ F₄₇₀(t) − F_405fit(t) ] / F_405fit(t). For the peri-event time histogram (PETH), the baseline value F₀ and standard deviation SD₀. was calculated using a −5 to −3 second window, except for the recordings of initial intruder encounters and first sniff of urine, where we used −30 to −20 second windows prior to each behavioral event. Overlapping behavioral bouts within this time window were excluded from the analysis. Then PETH was calculated by [ (F_n(t) − F₀ ] / SD₀. The peak and areas under the PETH curve were calculated within the 3 second window, except for initial encounters and first sniff of urine, where we used a 30 second window and a 10 second window after onset of behavior, respectively. We confirmed that the latency to achieve PETH peak level is shorter than the indicated time window.

Optogenetics

Before initiating behavioral experiments, the light intensity achieved at the tip of the optic fiber was estimated by connecting an equivalent optic fiber to the patch cable and measuring the light intensity at the tip of the fiber using a power meter. Laser power was controlled by turning an analogue knob on the laser power supply. Mice were connected to a 445 nm laser or 593 nm laser (Changchun New Industries Optoelectronics Tech) via optical patch cords (diameter 200 μm, N.A., 0.22, Doric lenses and Thorlabs) and a rotary joint (Doric lenses). Mice were allowed to habituate to the cables after connecting them for at least 5 minutes before starting behavior tests. The experimenter monitored mouse behavior via a computer monitor in a room adjacent to the behavioral arena and triggered the laser manually.

For α cell silencing (Figure S4A–E) and activation (Figure 2A–G) in virgins, we repeatedly performed photostimulation bout consisting of a sham stimulation (OFF) trial and light stimulation (ON) trial, for 30 minutes or up to male ejaculation. Bouts were separated by a ~30 second interval. The whole bout would be excluded if no male mounting attempt occurred during any of the two trials. For α cell activation (Figure 2H–K) and β cell silencing (Figure 3A–E) in lactating females and β cell activation in virgin (Figure 3F–P) and lactating females (Figure S4F–J), we alternately delivered sham stimulation (OFF) trials and light stimulation (ON) trials when animals were engaged in the behaviors of interest, during equivalent behavioral epochs, with at least a two-minute interval between trials. The detailed frequency and duration of photostimulation for each trial were indicated in the Figures and were controlled using an Accupulse Generator (World Precision Instruments) or an Isolated Pulse Stimulator (A-M Systems).

To determine the estrus phases of tested females, vaginal smear cytology was applied on the same day as the behavior test. A vaginal smear was collected immediately after the behavioral test and stained with 0.1% crystal violet solution for 1 minute. Cell types in the stained vaginal smear were checked microscopically. In this study, the estrus phase was characterized by many cornified epithelial cells and no leukocytes. Diestrus was characterized by many leukocytes.

Histology

Once the behavioral experiments were finished, virus expression and implant placement were histologically verified on all mice. Mice lacking correct virus expression or implant placement were excluded from the analysis. Mice were transcardially perfused with 1x PBS at room temperature, followed by 4% paraformaldehyde (PFA) (diluted from 16% EM grade PFA). Brains were extracted and post-fixed in 4% PFA 16–24h at 4°C, followed by 24 hours in 30% sucrose/PBS at 4 °C. Brains were embedded in OCT mounting medium, frozen on dry ice and stored at −80°C for subsequent sectioning. Brains were sectioned into 60–100 μm slices on a cryostat (Leica Biosystems). Sections were washed with 1× PBS and mounted on Superfrost slides, then incubated for 15 minutes at room temperature in DAPI/PBS (0.5 μg/ml) for counterstaining, washed again and coverslipped. Sections were imaged with an epifluorescent microscope (Olympus VS120)

QUANTIFICATION AND STATISTICAL ANALYSIS

All data were analyzed blind to relevant variables including estrous cycle, treatment, genotype (mice and virus), and light illumination conditions. Statistical analysis was performed using GraphPad PRISM (GraphPad Software). We first determined sample distribution normality using Kolmogorov–Smirnov tests. In experiments with paired samples, we used a paired t test and a Wilcoxon rank test for parametric and non-parametric data, respectively. In all other experiments, we used an unpaired t test and Mann-Whitney U-test for parametric and non-parametric data, respectively. Supplemental Table 1 states our sample sizes and definitions as well as precision measures.

10x Data analysis and clustering

All downstream analysis of Act-seq 10x scRNA-seq data was performed with R package Seurat (v3.1.3) (Stuart et al., 2019). The 10x gene expression matrices from all samples were combined as one Seurat object for analysis. We filtered cells with feature counts over 50000, UMIs less than 600, or mitochondrial UMIs exceeding 15%. For each remaining cell, gene expression was normalized by total number of counts, multiplied by a scale factor (10,000), and log-transformed (NormalizeData function). Then, 2000 highly variable genes were identified (FindVariableGenes function; top 2,000 genes with the highest standardized variance selected by selection.method = ‘vst’) after removing immediate early genes (e.g., Fos, Fosl2, Junb; 139 genes in total from (Wu et al., 2017)). The gene expression levels of those 2000 genes were scaled relative to the proportion of mitochondrial UMIs and total molecular counts of each cell (regressed out) (ScaleData function). Next, we applied principal component analysis (PCA) to reduce dimensionality; the first 40 PCs were used to generate clusters (FindNeighbors function; FindClusters function). Specifically, we performed iterative rounds of clustering and cell selection. For the first round, we distinguished neuronal cells from non-neuronal cells using marker genes (Stmn2 for neurons; Cldn5 for endothelial cells; C1qc for microglia; Opalin for oligodendrocytes; Gja1 for astrocytes; Pdgfra for OPCs; Mustn1 for mural cells). For the second round, we selected VMH cells using marker genes Slc17a6, Adcyap1, Fezf1, Dlk1 and Nr5a1 (reported in Kim et al., 2019). 19,103 VMH cells were clustered into 27 transcriptomic cell types (resolution 1.5). Clustering robustness for VMH clusters was evaluated by varying the resolution (1–2). Cluster identification was robust across the range of resolutions.

VMH transcriptomic data integration

To map female ventral VMH transcriptomic cell types onto the male dataset (Kim et al., 2019), we performed data integration using Seurat v3.1.3 (Stuart et al., 2019). Briefly, we selected the top 2,000 genes with the highest dispersion (variance to mean ratio) from each dataset (male-VMH-SMART-seq: 3,850 neurons; male-VMH-10x: 41,182 neurons; female-VMH-10x: 19,103 neurons) and took the union of these resulting gene sets. After removing unwanted genes (sex-specific, immediate early, retro-virus-induced, and noise-sensitive genes; as described in Kim et al., 2019), 3,049 genes were used as input to identify the pairwise correspondences between single cells across two different datasets, called anchors (FindIntegrationAnchors function; normalization.method = “LogNormalize”; first 75 PCs were used; male-VMH-10x was used as reference data during integration). We then used these anchors to integrate the datasets together (IntegrateData function; normalization.method = “LogNormalize”; first 75 PCs were used) and performed a joint clustering on these aligned embeddings (FindNeighbors function with first 75 PCs followed by FindClusters function with resolution 1.5), yielding 38 clusters (referred to as “Integrated cell types”; Figures S3A,B).

Supplementary Material

Video S1

Video S1. Activation of α cells promoted lordosis in virgin female. Related to Figure 2.

Video S2

Video S2. Activation of α cells inhibited attack in lactating dam. Related to Figure 2.

Video S3

Video S3. Activation of β cells evoked attack in virgin female. Related to Figure 3.

Key resources table.

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Bacterial and virus strains
AAV2-EF1a-DIO-hChR2(H134R)-EYFP	UNC vector core	N/A
AAV5-EF1a-DIO-eNpHR3-EYFP	UNC vector core; Gradinaru et al., 2010	Addgene# 26966
AAVDJ-hSyn-Coff/Fon-hChR2(H134R)-EYFP	UNC vector core; Fenno et al., 2014	Addgene# 55648
AAVDJ-Syn-Coff/Fon-EYFP	UNC vector core; Fenno et al., 2014	Addgene# 55652
AAV1-Syn-Flex-GCaMP6m	Addgene; Chen et al., 2013	Addgene# 100838
AAV8-nEF-Coff/Fon-eNpHR3.3-EYFP	Addgene; Fenno et al., 2020	Addgene# 137154
AAVDJ-EF1a-Coff/Fon-GCaMP6m	Addgene; Fenno et al., 2020	Addgene# 137121
Chemicals, peptides, and recombinant proteins
Papain	Sigma	Cat# P3125–250MG
Egg White/BSA ovomucoid inhibitor	Worthington	Cat# OI-BSA
DNase I	Thermo scientific	Cat# 90083
DNase/RNase-Free Distilled Water	Thermo scientific	Cat# 10977–015
DAPI	Sigma	Cat# D9542–10MG
BSA	Thermo scientific	Cat# AM2616
Actinomycin D	Sigma	Cat# A1410–2MG
Kynurenic acid sodium salt L-Cysteine	Abcam	Cat# ab120256
L-Cysteine	Sigma	Cat# W326305–100G
D-(+)-Trehalose dihydrate	Sigma	Cat# 90210–50G
Ethylenediaminetetraacetic acid	Sigma	Cat# 03690–100ML
MgSO4 solution	Sigma	Cat# 83266–100ML-F
CaCl2 solution	Sigma	Cat# 21115–100ML
Sodium nitrite	Sigma	Cat# 237213–100G
Heparin sodium salt	Sigma	Cat# H4784–250MG
Paraformaldehyde	EMS	Cat# 15714-S
Sucrose, RNase & DNase Free	Amresco	Cat# 0335–2.5KG
Critical commercial assays
Chromium Next GEM Single Cell 3’ GEM, Library & Gel Bead Kit v3.1	10x Genomics	PN-1000121
Chromium Next GEM Chip G Single Cell Kit	10x Genomics	PN-1000120
Chromium i7 Multiplex Kit	10x Genomics	PN-120262
Experimental models: Organisms/strains
Mouse: Esr1-flpo	Karigo et al., 2020	N/A
Mouse: Esr1-cre	Lee et al., 2014	N/A
Mouse: Npy2r-cre	Chang et al., 2015	JAX No. 029285
Mouse: C57BL/6N	Charles River	N/A
Mouse: BALB/c	Charles River	N/A
Software and algorithms
CellRanger v3	10x Genomics	http://www.10xgenomics.com/
Python v3	Python Software Foundation	http://www.python.org/
R v3.6.2	R Foundation	R Foundation https://www.r-project.org/
Seurat v3	Stuart et al., 2019	https://satijalab.org/seurat/
10x analysis code (R and python)	This paper	N/A
Photometry analysis code (python)	This paper	N/A
Prism 6	GraphPad Software	https://www.graphpad.com/
ImageJ	NIH	https://imagej.nih.gov/ij
10x Seq data	This paper	Mendeley DOI: 10.17632/w25zm9cn6x.1

Highlight:

• scRNAseq reveals distinct cell types in female VMHvl active in mating vs. aggression

• Activation of VMHvl^{Esr1+,Npy2r−} (α) cells promotes mating behaviors in virgins

• Activation of VMHvl^Npy2r+ (β) cells promotes aggression in both virgins and dams

• Response of β but not α cells to male social cues increases in lactating dams