Serotonin modulates nucleus accumbens circuits to suppress aggression

Abstract

Serotonin (5-hydroxytryptamine; 5HT) has long been considered anti-aggressive, but the mechanisms by which 5HT regulates downstream circuits to control aggression remain unclear. Combining fiber photometry, optogenetics, and miniaturized microscope recordings in double-transgenic mice, we find that 5HT levels ramp up in the nucleus accumbens during aggression, inhibiting a subset of D1 medium spiny neurons to suppress attacks. Our results reveal a novel 5HT-mediated neuromodulatory mechanism for limiting aggressive behavior.

Introduction

Low levels of brain serotonin (5HT) have been associated with heightened aggression across the animal kingdom^1,2. Drugs boosting the 5HT system, however, produce variable effects on aggression³, forcing clinicians to prescribe agents that sedate patients rather than specifically reduce aggression^4,5. The complex behavioral effects of brain-wide 5HT modulation may be due to heterogeneity in the 5HT system, with distinct 5HT neuron populations projecting to different targets to mediate diverse behaviors^6–8. While recent studies have begun to map specific 5HT targets onto their associated behaviors^8–11, little is known about the target regions that control aggression, or how 5HT modulates neural activity in these targets^12,13.

The nucleus accumbens (NAc) has long been identified as a key node controlling motivated behaviors^14,15. Although most research has focused on the role of dopamine (DA) in this region¹⁶, recent studies have implicated NAc 5HT release in promoting a range of prosocial behaviors, including behavioral antecedents of empathy^17–20. In particular, the medial shell subregion of the NAc, the site with the most abundant aggression-activated neurons²¹, is a major target of 5HT axons projecting from the dorsal raphe (DR)¹⁰. We therefore hypothesized that 5HT release in the NAc medial shell may also influence aggression, and that it would do so via distinct mechanisms from those engaged by DA.

Results

5HT and DA levels in the NAc increase during aggression with distinct temporal profiles

To measure real-time changes in 5HT and DA levels in the NAc during aggression, we used fiber photometry to record either GRAB-5HT²² or GRAB-DA²³ signals in the NAc during a resident-intruder assay²⁴, where a weak intruder mouse was introduced to the home cage of the test mouse (Fig. 1a, b; Supplementary Fig. 1a, b). We found that 5HT levels were low at aggressive approach, ramped up during the attack itself, and peaked around aggression offset (Fig. 1c–e). While the peak and area under the curve (AUC) positively correlated with aggression duration (Supplementary Fig. 2), the slope of the 5HT ramp showed a strong negative correlation (Fig. 1f): the faster the rise in 5HT, the shorter the aggressive bout. In contrast, DA levels rose during both the pre-attack approach period and at attack onset (Supplementary Fig. 1c–e), and only weakly correlated with aggression duration (Supplementary Fig. 1f). These data suggest that DA and 5HT may signal different phases of aggression, with 5HT selectively engaged in mediating the termination of attacks.

Figure 1: 5HT signaling in the NAc during aggression.

(a) GRAB-5HT virus was injected in the NAc for fiber photometry recording. (b) Example session of GRAB-5HT recording during a resident-intruder test. (c) Average GRAB-5HT (green) and control (grey) signal aligned to aggressive approach, attack, or aggression offset. (d) GRAB-5HT signal averaged in a 2-second window pre/post the behavioral events as marked in (c) (left to right: n.s. P = 0.28, 0.21, *P = 0.031, **P = 0.006, 0.001, one-sample t-test, n = 6 mice). (e) Heatmap showing GRAB-5HT signaling in individual aggressive bouts sorted by duration. (f) The duration of aggressive episodes negatively correlates with the slope of 5HT increase (***P<0.0001, r = −0.47, Pearson’s correlation coefficient). (g) Either saline or MDMA (7.5 mg/kg) was injected intraperitoneally 15 min before experiments. (h) Frequency of attacks in the resident-intruder test after injection of MDMA or saline (**P = 0.0039, Wilcoxon signed-rank test, n = 9 mice). (i) Flp-dependent excitatory opsin was injected into the DR of SERT-Flp mice. A bilateral optical fiber was implanted in the NAc for optogenetic stimulation of 5HT terminals. (j) Change in the frequency of attacks in photostimulation compared with non-photostimulation sessions (n.s. P = 0.19, Wilcoxon signed-rank test, n = 11 SERT-Flp mice). (k) Same as (j) but for the change in average duration of attack bouts (**P = 0.0020, Wilcoxon signed-rank test).

5HT but not DA release in the NAc suppresses aggression

We next tested the causal roles of 5HT or DA release in aggressive behavior. We initially examined the effects of (±)3,4-methylenedioxymethamphetamine (MDMA), a translationally-relevant compound that enhances both 5HT and DA release but promotes prosocial behaviors primarily by increasing 5HT levels in the NAc^18,25. Consistent with prior studies²⁶, administration of MDMA (7.5 mg/kg, i.p.) to the resident mouse 15 min before the resident-intruder test completely abolished aggression, compared to a saline control (Fig. 1g, h). Although this finding suggests that increased 5HT release may reduce aggression, the drug has numerous effects throughout the brain, and cannot be interpreted as selective to either 5HT or the NAc.

To more specifically manipulate either 5HT or DA release in the NAc, we injected the Flp-dependent excitatory opsin, FLPX-ChRimsonR or fDIO-rsChRmine, in either the DR of SERT-Flp mice (Fig. 1i), or the ventral tegmental area of DAT-Flp mice, and implanted optical fibers in the NAc (Supplementary Fig. 1g). Optogenetic activation of DA inputs in the NAc during the resident-intruder test did not affect either the frequency of attacks or the duration of each attack bout (Supplementary Fig. 1h, i), although we validated that photostimulation did trigger DA release (Supplementary Fig. 1j–l). In contrast, 5HT input activation significantly reduced the duration of attacks, without changing attack frequency (Fig. 1j, k, Supplementary Fig. 3) or basic locomotor, social, or valence-related behaviors (Supplementary Fig. 4).

To begin to determine the 5HT receptors mediating the anti-aggressive properties of NAc 5HT release, we infused a variety of 5HT receptor-specific agonists into the NAc immediately prior to the resident-intruder assay. Consistent with previous results examining the prosocial effects of NAc 5HT release¹⁷, infusion of a 5HT1b receptor agonist (CP93129) significantly reduced aggression, while 5HT1f and 5HT2c receptor agonists did not (Supplementary Fig. 5). These results suggest that in the medial NAc shell, 5HT release, but not DA release, is sufficient to reduce aggressive behavior, with a particular role in terminating attack bouts.

D1 but not D2 MSNs in the NAc promote aggression

What downstream cells mediate the effects of NAc 5HT on aggression? The vast majority of NAc neurons are medium spiny neurons (MSNs) expressing one of two DA receptors: the D1 receptor, which is thought to augment neural activity in response to DA; or the D2 receptor, which does the opposite²⁷. In the context of aggressive behavior, D1 MSNs in the NAc have been suggested to signal the rewarding aspect of aggression^21,28. However, optogenetic stimulation of DA axons in the NAc, which presumably activates D1 MSNs, failed to affect aggression in prior studies^29,30.

To determine how D1 and D2 MSNs are engaged during aggression, we injected the Cre-dependent calcium indicator, FLEX-GCaMP8m, in either D1-Cre or A2A-Cre mice, and recorded the activity of D1 or D2 MSNs during the resident-intruder test using a miniature microscope (Fig. 2a, b). On average, both MSN cell types exhibited increased activity during aggression (Supplementary Fig. 6a–d), but significantly more D1 MSNs than D2 MSNs showed robust activation across attacks (Fig. 2c, d). Examining individual trials, we found that the activity of D1 MSNs, but not D2 MSNs, positively correlated with the duration of aggressive episodes (Fig. 2e, f). This suggests that D1 MSN activity, like 5HT release, may influence the length of aggressive episodes. To test this hypothesis, we injected the Cre-dependent inhibitory opsin, NpHR3.0 in the NAc of either D1-Cre or A2A-Cre mice to optogenetically inhibit either D1 or D2 MSNs during the resident-intruder test (Fig. 2g). Similar to the effect of 5HT input activation, D1 MSN (but not D2 MSN) inhibition reduced the duration of attack bouts without influencing attack frequency (Fig. 2h, i; Supplementary Fig. 3c–f) or other locomotor, social, or valence-related behaviors (Supplementary Fig. 6e–j).

Figure 2: D1 and D2 MSN signaling in the NAc during aggression.

(a) FLEX-GCaMP8m virus was injected in the NAc of D1-Cre or A2A-Cre mice for calcium imaging with a miniscope. (b) Example field-of-view showing detected neurons during the resident-intruder test (scale bar, 100μm). (c) Example calcium traces of cells robustly activated at attacks. (d) Percentage of D1 and D2 MSNs significantly activated during attacks (**P = 0.0011, Wilcoxon rank-sum test). (e) Correlation between the duration of the aggression episode and the average D1 MSN activity during aggression episodes (****P < 0.0001, r = 0.29, Pearson’s correlation coefficient). (f) Same as (e) but for D2 MSNs (n.s. P = 0.23). (g) Cre-dependent inhibitory opsin or control fluorophore (DIO-NpHR3.0 or DIO-mcherry) was injected in bilateral NAc of D1-Cre or A2A-Cre mice for optogenetic stimulation. (h) Change in the frequency of attacks in photostimulation compared with non-photostimulation sessions (n.s. P = 0.99, 0.96, one-way ANOVA followed by Tukey-Kramer test, n = 9 mice for D1 MSN inhibition, 7 mice for D2 MSN inhibition and 7 mice for controls). (i) Same as (h) but for the change in the average duration of attack bouts (n.s. P = 0.79, *P = 0.019, one-way ANOVA followed by Tukey-Kramer test).

5HT release in the NAc differentially modulates D1 and D2 MSNs

Given that 5HT stimulation and D1 MSN inhibition in the NAc produce similar anti-aggressive behavioral effects, we hypothesized that 5HT may specifically reduce the activity of D1 MSNs to suppress attack duration. To test this prediction, we first measured D1 or D2 MSN GCaMP8m activity using a miniscope when the mice were administered the same dose of MDMA (7.5 mg/kg) that abolished aggression in the resident-intruder tests (Fig. 3a). We found that MDMA administration led to a significant decrease in D1, but not D2 MSN firing (Fig. 3b–d). This result is consistent with our hypothesis that 5HT inhibits D1 MSN activity, but again does not control for other possible network effects of MDMA administration.

Figure 3: 5HT preferentially inhibits D1 MSNs that signal aggression.

(a) Experimental strategy: miniscope recording of either D1 or D2 MSNs expressing FLEX-GCaMP8m in the NAc after the mouse was injected with either saline or MDMA (7.5 mg/kg). (b) Change in the inferred spike rate in D1 MSNs after saline or MDMA injection. Grey areas indicate injection time. (c) Same as (b) but for D2 MSNs. (d) Change in the average inferred spike rate in D1 and D2 MSNs post MDMA injection compared to baseline (**P = 0.0037, Wilcoxon rank-sum test, n = 8 D1-Cre mice and 7 A2A-Cre mice). (e) D1-Cre or A2A-Cre mice were crossed with SERT-Flp mice to create double transgenic D1-Cre × SERT-Flp or A2A-Cre × SERT-Flp mice. (f) Double-transgenic mice in (e) were injected with Cre-dependent GCaMP8m virus in the NAc and Flp-dependent ChRimsonR in the DR. (g) Example brain slice showing the expression of GCaMP8m in NAc neurons and ChrimsonR in the terminals of 5HT neurons. (h) Validation experiment: SERT-Flp mice were injected with Flp-dependent ChRimsonR in the DR and GRAB-5HT in the NAc and an optical fiber was implanted in the NAc to record 5HT signals during photostimulation of 5HT terminals (n = 2 mice: 1 SERT-Flp × D1-Cre and 1 SERT-Flp × A2A-Cre). (i) Calcium activity of D1 MSNs aligned with photostimulation (30 s, 20 Hz, n = 6 D1-Cre, SERT-Flp mice). (j) Same as (i) but for D2 MSNs (n = 6 A2A-Cre, SERT-Flp mice). (k) Average calcium response to photostimulation in D1 and D2 MSNs (**P = 0.0022, Wilcoxon rank-sum test). (l) Percentage of cells robustly activated or inhibited across trials (n.s. P = 0.33, 0.86, **P = 0.0032, ***P = 0.00037, two-way ANOVA followed by Tukey-Kramer test). (m) Calcium activity of aggression-modulated D1 MSNs around aggressive approach. Each row is a cell, sorted by the calcium response in time 0–2 s (n = 344 cells, 12 sessions, 6 mice). (n) Calcium activity of the same cells in (m) aligned with photostimulation, which was delivered later that same day. (o) Photostimulation response of cells sorted by aggression response as in (m-n). (top to bottom: ** P = 0.007, *P = 0.035, 0.031, n.s. P = 0.35, 0.84, one-sample t-test).

To more specifically assess the effects of endogenous 5HT release on MSN activity, we generated double-transgenic mice by crossing D1-Cre or A2A-Cre mice with SERT-Flp mice, and injected them with Cre-dependent GCaMP8m virus in the NAc and Flp-dependent ChRimsonR virus in the DR (Fig. 3e, f). Placement of a GRIN lens over the NAc injection site allowed us to simultaneously stimulate 5HT release and record genetically-identified downstream neurons while mice rested in their home cages (Fig. 3g). Consistent with the MDMA results, optogenetic stimulation of 5HT inputs in the NAc (Fig. 3h) overall inhibited the activity of D1 MSNs, but not D2 MSNs (Fig. 3i–l).

5HT preferentially inhibits MSNs that signal aggression in the NAc

Next, we took advantage of our cellular-resolution recording technique to determine which D1 MSNs were particularly sensitive to 5HT-mediated inhibition. In two sessions on the same day, we first recorded D1 MSNs during a resident-intruder task, and later recorded the same neurons during optogenetic stimulation of 5HT release. We sorted aggression-modulated D1 MSNs (~37% of all D1 MSNs) by the magnitude of their GCaMP8m activity at aggression onset (Fig. 3m), and then plotted their response to later stimulation of 5HT inputs (Fig. 3n). We found that the D1 MSNs that were more active during aggressive behavior were also more inhibited by 5HT release (Fig. 3o). In contrast, when we repeated the same experiments with D1-Cre/DAT-Flp mice (Supplementary Fig. 7a–c), we found that DA release predominantly excited D1 MSNs, with no preferential effects on aggression-modulated neurons (Supplementary Fig. 7d–f). Taken together, our results imply that 5HT, but not DA, selectively inhibits a subset of aggression-active D1 MSNs to suppress attacks.

So far we have measured the activity of the same D1 MSNs in two separate sessions: first in a resident-intruder paradigm, and later with optogenetic stimulation of 5HT inputs. However, it is possible that the effect of 5HT on D1 MSN activity differs depending on context. Thus, we ran animals through a third session that combined optogenetic stimulation with the resident-intruder paradigm, allowing us to examine the effect of optogenetically-triggered 5HT release on D1 MSNs during aggression itself (Supplementary Fig. 8). Although 5HT release did not change the fraction of D1 MSNs that were activated at aggressive approaches (Supplementary Fig. 8d), it significantly suppressed their activity after attack onset (Supplementary Fig. 8e, f).

Discussion

Previous microdialysis measurements of 5HT levels did not have the temporal resolution to capture the sub-second dynamics of 5HT release during aggressive behavior³¹. Here we utilized the latest ‘GRAB’ sensors and optogenetics to reveal that NAc 5HT levels ramp up during aggressive episodes, peaking at aggression offset and helping to terminate attacks. We then combined these manipulations with cellular-level recordings in the NAc, discovering the downstream mechanism of 5HT’s behavioral effect; namely, that 5HT release inhibits a subset of aggression-activated D1 MSNs.

Despite a long history of cross-species work implying that 5HT is anti-aggressive^1,2, prior studies stimulating all DR neurons or inhibiting all 5HT neurons found little to no acute effect on aggression^32,33. Our results suggest that DR 5HT neurons may influence aggression in a projection-specific manner, which in this case includes the NAc. Moreover, we found that endogenous NAc 5HT release primarily affected the duration, but not the frequency, of attack bouts. This finding suggests that the NAc may play a specific role in maintaining, rather than initiating, attacks. Closed-loop optogenetics will be an advantageous approach to investigate the temporal specificity and necessity of the DR 5HT-to NAc projection in mediating different phases of aggressive behavior^34–37.

Using in vivo calcium imaging to measure the activity of individual D1 and D2 MSNs, we found that although both populations were activated during aggression, only D1 MSN activity correlated with aggression duration on a trial-by-trial basis. Furthermore, D1, but not D2 MSN inhibition, modulated aggressive behavior. The crucial role for D1 MSNs in aggression was also supported by our experiments in double-transgenic mice, where we stimulated 5HT release and simultaneously recorded the effects of this release on downstream MSN activity. Despite the broad axonal projections and diverse behavioral functions of 5HT neurons³⁸, little was known regarding the neuronal response to 5HT release in target areas^8,39. We found that optogenetically-evoked 5HT release selectively suppressed the activity of D1 MSNs, in particular those that were normally activated during aggression. Furthermore, by recording and stimulating at the same time during the resident-intruder assay, we confirmed that 5HT dramatically suppressed the activity of the aggression-specific D1 neurons. These findings extend previous cFos and chemogenetic results implicating D1 MSNs in aggression reward²¹, adding new evidence supporting the selective engagement of medial shell NAc D1 MSNs in aggressive behavior^21,28,40. Note, however, that we cannot rule out the possibility that D1 MSN inhibition was triggered by co-release of GABA from 5HT terminals. DR serotonin neurons can co-express gene markers associated with glutamatergic or GABAergic transmission, or both^41,42. Future work should test whether neurotransmitter or neuropeptide co-release contributes to the neural and behavioral effects observed in our study.

Although we focused much of our work on selective optogenetic stimulation of 5HT inputs, we also performed studies using MDMA, a clinically-relevant, pro-serotonergic compound thought to enhance sociability¹⁸. We found that systemic injection of MDMA abolished aggressive behavior and preferentially inhibited D1 MSN calcium activity, consistent with both prior behavioral studies²⁶ and our own experiments manipulating D1 and D2 MSNs. Previous ex vivo slice experiments found that bath application of MDMA reduced excitatory postsynaptic currents (EPSCs) in both D1 and D2 MSNs in NAc slices¹⁸. The difference between these prior results and the current study are likely due to the myriad differences between in vitro and in vivo preparations, including the presence versus absence of long-range inputs to the NAc; the 5HT- and DA-releasing potential of MDMA in slices versus intact brains; the precise NAc region that was targeted for recordings; and the differences between recording of evoked EPSCs versus activity-dependent intracellular calcium levels.

Consistent with previous recording^31,43 and manipulation studies^29,30, DA release in the NAc increased during aggression but optogenetic stimulation of DA inputs did not modulate aggressive behavior. How did DA release not modulate aggression even though it caused an overall increase in D1 MSN activity? One important difference between the actions of DA and 5HT was that 5HT preferentially inhibited the aggression-active D1 MSNs while DA’s effects did not exhibit such specificity. Thus, the action of DA may leave the relative strength of the aggression-related MSN ensemble unchanged. The mechanisms by which 5HT exerts its inhibitory effect on D1 MSNs remain unknown but likely rely on both presynaptic and postsynaptic 5HT receptors. Presynaptically, we found that infusion of a 5HT1b agonist into the NAc suppressed aggression, presumably in part by suppressing excitatory transmission from glutamatergic inputs to the NAc^17,44. Postsynaptically, we recently reported that NAc D1 and D2 MSNs express different complements of 5HT receptors⁴⁵ with D1 MSNs expressing higher levels of Gi-coupled (presumably inhibitory) 5HT receptors, while D2 MSNs express more Gq/Gs-coupled receptors. Thus, together, the pre-and postsynaptic actions of 5HT would be expected to inhibit D1 MSNs.

Previous work has shown that 5HT release in the NAc promotes social behavior, as demonstrated by juvenile interaction and three-chamber sociability tests^17,18. The present findings extend this work by showing that NAc 5HT release also suppresses aggressive behavior through a previously unrecognized mechanism. It is important to note, however, that NAc 5HT release can play multiple other roles. In simple reinforcement learning, for example, NAc 5HT can act in opposition to DA¹⁰. These observations highlight that the behavioral roles of neuromodulators like 5HT and DA, even within the same structure such as the NAc, are multifaceted and cannot be restricted to single behavioral categories. Nevertheless, by understanding the molecular, cellular, and circuit mechanisms through which 5HT release exerts its prosocial and anti-aggressive effects, we may uncover novel therapeutic strategies for clinical conditions where aggression is a major concern.

Methods

Mice

Wild-type C57BL6/J (Jackson Laboratory #000664), stock number: 017260-UCD), SERT-Flp (Jackson Laboratory #034050), DAT-Flp (Jackson Laboratory #035436), D1-Cre (MMRRC, #029178-UCD) and A2A-Cre (MMRRC, #036158-UCD) mice were used for breeding SERT-Flp^+/−, D1-Cre^+/−, A2A-Cre^+/−, SERT-Flp^+/− × D1-Cre^+/−, SERT-Flp^+/− × A2A-Cre^+/− and DAT-Flp^+/− × D1-Cre^+/− mice for neural recording and manipulation. Adult male BALB/c mice (> 8 weeks old, group-housed, 3–5 per cage) were used as intruders into the home cages of male resident mice (> 8 weeks old, singly housed). All experimental mice were housed on a 12-h light/ dark cycle with food and water ad libitum. Behavioral Tests were run in the dark phase. All procedures complied with the animal care standards set forth by the National Institute of Health and were approved by Stanford University’s Administrative Panel on Laboratory Animal Care.

Surgeries

General surgery procedure: mice (8–12 weeks old, male) were anaesthetized with isoflurane (5% induction, 1.5% maintenance) for surgeries. A stereotaxic frame (Kopf instruments) was used to target injections and implantations (AP and ML are relative to bregma and DV relative to the skull surface). The coordinates used for the NAc were the following in all experiments: AP: +1, ML: 0.7, DV: −4.4 mm. Viruses were infused with a syringe pump (Harvard Apparatus) at a rate of 150 nl/min and allowed to diffuse for at least 5 min. Implants were secured to the skull using screws (Antrin Miniature Specialties) with dental cement (Super-Bond C&B, Sun-Medica and Dual-Cure Resin- ionomer, Geristore). Optical implant placements were histologically verified post-hoc, and mice with mistargeted fibers or lenses were excluded from further analysis. Below lists the virus, target coordinates and implant details specific for different experiments.

For optogenetic stimulation of DR or ventral tegmental area (VTA) terminals, 1μl AAV8-Syn-FLPX-rc [ChRimsonR-tdTomato] (Addgene) or AAV8-Ef1a-fDIO-rsChRmine-oScarlet-WPRE (Stanford viral core) was injected in the DR of SERT-Flp mice, or 0.5 ul of the same virus was injected in bilateral ventral tegmental area of DAT-Flp mice. rsChRmine was used in the optogenetic experiments for its high sensitivity⁴⁶.

For optogenetic inhibition of D1 or D2 MSNs, 700 nl of AAVdj-EF1a-DIO-NpHR3.0-mcherry or AAVdj-Ef1a-DIO-mcherry (Stanford viral core) was injected bilaterally in the NAc of D1-Cre or A2A-Cre mice. Bilateral optical fibers (200–400 μm diameter, Doric) were implanted over the NAc.

For fiber photometry recordings of 5HT or DA, 700 nl of either AAV9-hSyn-5HT3.0 or AAV9-hSyn-GRABDA2m (WZ Biosciences) was injected into the NAc. An optical fiber (400 μm diameter, Doric) was implanted over the NAc for photometry recording.

For ‘Miniscope’ experiments, SERT-Flp × D1-Cre, SERT-Flp × A2A-Cre or DAT-Flp × D1-Cre mice were injected with 1μl of AAV9-syn-FLEX-jGCaMP8m-WPRE (1:2 dilution with phosphate-buffered saline (PBS), Addgene) in the NAc, together with 1μl of AAV8-Syn-FLPX-rc [ChRimsonR-tdTomato] (Addgene) injected in either the DR (AP: −4.6, ML: 0, DV: −3 mm) of SERT-Flp mice or the ventral tegmental area (AP: −3.4, ML: 0.2, DV: −4.25 mm) of DAT-Flp mice. We chose ChRimsonR for the miniscope experiments to minimize potential crosstalk between the blue LED (435–460 nm wavelength for calcium imaging) and optogenetic stimulation⁴⁷. A gradient index (‘GRIN’) lens (0.6 mm in diameter, 7.3 mm long, ProViewTM Integrated Lens, Inscopix) was implanted at the injection site in the NAc.

For drug microinjection, bilateral cannulae (P1 technologies) were implanted into the NAc of mice of wild-type background.

Fiber photometry

Mice with fiber implants were connected through low-autofluorescence patch cords and a dichroic cube (Doric #FMC6) to a photodetector (Newport #2151). GRAB sensors were excited by frequency-modulated LED light of 405 and 465 nm for near-isosbestic (used as control) and activity-dependent recordings, respectively. The resulting fluorescence signals were demodulated and recorded using a signal processor (RZ5P from Tucker-Davis Technologies running Synapse software) sampled at 1.0173 KHz. For experiments combining fiber photometry with optogenetic stimulation, 9–10 mW, 20 Hz pulses of red LED light (5 ms per pulse) were delivered in either 30-s trains (10 trains, each separated by 90 s; Fig. 3h, Supplementary Fig. 1k) or one 3-min train (Supplementary Fig. 1l). Raw data was converted into .mat files using software API from Tucker-Davis Technologies (TDT). For the aggression sessions, raw traces were down-sampled by 3x and de-bleached⁴⁸ to get session-wise zscored fluorescence (Fzsc) which was then cropped and aligned to behavioral events (aggression approaches, attacks etc.).

‘Miniscope’ imaging and optogenetic stimulation

Calcium imaging and photostimulation were performed using a miniature microscope (nVoke 2.0, Inscopix)⁴⁷. Blue LED light (0.5–1 mW, 435–460 nm) was used for imaging at 10 Hz per imaging plane (512 × 512 pixels per plane). Two planes separated by ~150 μm were imaged near-simultaneously.

For imaging sessions with MDMA/saline injections, mice were habituated to intraperitoneal injections a day before experiments. On a day of experiment, we recorded baseline activity for 5 min, administered either MDMA (7.5 mg/kg, Sigma-Aldrich) or saline intraperitoneally, waited 5 min, and then recorded for another 20 min. The sequence of MDMA or saline injection was counter-balanced across mice. Injections in the same mice were separated by at least 48 hours.

For imaging sessions during resident-intruder tests and/or optogenetic stimulation, three recordings were completed on the same day with the same field-of-view. The first recording was performed during the resident-intruder test, without any photostimulation. In the second recording, 3 min of photostimulation (12 mW, 20 Hz) was delivered in the middle of a 9-minute resident-intruder assay (with a different intruder). In the third recording, 10 bouts of photostimulation were delivered while the animal was alone in its home cage; these bouts consisted of 30 s, 12 mW, 20 Hz pulses of red LED light (590–650 nm, 5 ms per pulse) separated by 90 s. The order of the first and second recordings was counterbalanced across days, and each mouse experienced two days of experiments (Supplementary Fig. 8a).

‘Miniscope’ imaging data analysis

Raw imaging data was converted into H5 data files using software from Inscopix (Bruker). Sessions recorded on the same day from the same field-of-view were concatenated before motion correction and neural signal extraction. Piecewise rigid motion correction was applied to each imaging plane⁴⁹. Neuronal shapes, denoised calcium activity traces, and inferred spikes were extracted by a constrained nonnegative matrix factorization (CNMF) method developed for endoscopic datasets⁵⁰. To avoid double counting the neurons that were captured in both imaging planes, the detected ROIs across planes that were spatially close (centroids < 10 pixels apart) and also had high activity correlation (>0.5) were merged into one. The calcium traces were z-scored for each session and aligned to either behavior timestamps (aggressive approach, attack etc.) or photostimulation onsets. These z-scores were then baseline-corrected to produce ΔFzsc, with the baseline defined as either the 1 s window starting 2 s before the behavior, or the 10 s window before photostimulation onset. Timestamps within 3 s of a prior behavioral event were ignored to avoid confounding baseline subtraction with behavior-related activity.

To identify aggression-modulated neurons, the aggression-aligned activity (averaged in a 2 s window following each aggressive approach) was compared to a distribution of activity values aligned to 1000 random timestamps. A neuron was considered to be aggression-modulated if its aggression-aligned activity was higher or lower than 90% of the random-timestamp-aligned activity values. For sessions with drug or saline injections, the inferred spike rate from each neuron was normalized by that during the baseline period.

Optogenetic manipulations during resident-intruder

For optogenetic manipulations, mice with optical implants were connected to a 625 nm LED light source (Prizmatix) via a plastic fiber (1 mm diameter, NA 0.63) and a fiber optic rotary joint (Doric). For photostimulation of DA or 5HT inputs to the NAc, mice underwent two days of optogenetics experiments. On each day, they performed two 9-minute sessions separated by >1h, one with photostimulation and one without. The order of photostimulation was counterbalanced between days. The photostimulation (3 min at 20 Hz, 12 mW, 5 ms pulse width) was initiated 3 min after intruder entry. For photoinhibition of D1 or D2 MSNs in the NAc, mice likewise underwent two sessions, separated by >1 hour. In the second session, we delivered a 9-minute train of 0.4 Hz, 8 mW, 2 s light pulses. The duration of optogenetic terminal stimulation was chosen to balance two factors: it had to be long enough to capture enough aggressive bouts to test the behavioral effect, but short enough to avoid opsin desensitization. Similar durations of 5HT terminal stimulation (5-min for three-chamber sociability and 2-min for juvenile interaction) produced pro-social effects¹⁷, so we hypothesized that 3-minute elevations of 5HT levels in the NAc would similarly influence attacking behavior. Video recordings and optogenetic stimulations were synchronized by TTL triggers from a data acquisition card (USB6259, National Instrument) controlled by PackIO⁵¹.

Drug injections

For intracranial drug microinjection, two 9-minute sessions were performed, separated by >1 h. 15 min before the second session. On one test day, after the first session. mice received a bilateral intra-NAc infusion of either saline with 1% DMSO, or one of the selective 5HT receptor agonists: 0.5 μg CP93129 diluted in saline targeting 5HT1b receptors; 1 μg LY344864 diluted in 1% DMSO targeting 5HT1f receptors, or 0.5 μg WAY163909 diluted in saline targeting on 5HT2c receptors. Drugs were infused through an injector cannula using a microinfusion pump (Harvard Apparatus) at a continuous rate of 0.1 μl per min to a total volume of 0.3 μl per hemisphere. Injector cannulae were removed 2 min after infusions were complete. Injections in the same mice were separated by at least 48 hours.

For systemic MDMA/saline injections, mice were habituated to intraperitoneal injections a day before experiments. On one day. either MDMA (7.5 mg/kg, Sigma-Aldrich) or saline was administered intraperitoneally 15 min before the behavioral tests. The sequence of MDMA or saline injection was counter-balanced across mice. Injections in the same mice were separated by at least 48 hours.

Resident-intruder tests

Mice were habituated to the recording arena (with fiber connection for optogenetic experiments) by performing a daily resident-intruder test until they attacked the intruder within 5 min after the intruder was introduced, and more than 2 times in 9 min on two consecutive days. Mice for microinjection experiments underwent a sham injection handling before these habituation sessions. 32 mice that did not reach the aggression criteria after 1 week were excluded from further experiments. On rare occasions when an intruder was aggressive towards the resident, the experiment was immediately aborted, and the intruder was not used for further experiments. For each test mouse, a different intruder was used for control (saline or no-light) and test (drug-treated or light-on) conditions. All resident-intruder sessions were 9-min long. We did not observe any injuries to the intruder that would require capping the number of bites or ending the session early. For detailed descriptions of how we combined resident-intruder tests with imaging or manipulations, please see the relevant sections above.

Resident-intruder behavior analysis

Behavioral chambers were illuminated with dim red light and videos were taken by an overhead camera (FLIR Blackfly) at 20 Hz frame rate in SpinView software (Teledyne Vision Solutions). Mouse postures were extracted from behavior video recordings using ‘DeepLabCut’⁵². For sessions with neural activity recordings, to get the exact onset time of attacks, videos were manually labelled by experimenters who were blinded to experimental condition or simultaneously-collected imaging data. To obtain more objective timestamps of behaviorally-relevant epochs in our neural data, we defined the onsets/offsets of aggression episodes (i.e. the ‘aggressive approaches’ and ‘aggression offsets’) as the time points when the two mice became close to (< 5 cm) or far away from (> 5 cm) each other before/after attacks. To reduce confounds in our baselining, we only analyzed bouts that lasted at least 2 s and were separated by at least 2 s from a prior bout. For sessions without neural activity recordings, we used a machine learning based software, ‘SIMBA’⁵³, to identify attacking behavior (Supplementary Notes, Supplementary Video 1). The detected attacking bouts that were < 0.25 s apart were merged into one, and those that were shorter than 0.5 s were filtered out. The behavioral metrics (duration per attack, sniffs per minute etc.) under LightOff and LightOn conditions were divided by the corresponding value in the LightOff condition to get a percentage change (%).

Open field test

Mice were placed into a 40 × 40 cm behavioral arena and allowed to explore freely for 12 mins. Optogenetic manipulations were delivered in four alternating light-on (0.4 Hz, 8 mW, 2 s light pulses for NpHR3.0 stimulation) and light-off epochs that lasted 3 mins each. Epoch order was counterbalanced across mice. Biobserve Viewer behavioral tracking program was used to extract distance traveled during each epoch.

Real-time place preference

Real-time place preference (RTPP) tests were performed in an arena with three chambers as previously described¹⁰: a neutral center chamber with clear floors, and two identical side chambers with white walls and floors. Mice were confined to the center chamber for a ~2 min habituation period. Then, the barriers were removed to begin the first half of the test during which mice were free to explore all three chambers for 15 min and one side chamber (left or right, counterbalanced) was paired with photostimulation (20 Hz, 12 mW, 5 ms pulses for terminal ChRimsonR or rsChRmine stimulation, or 0.4 Hz, 8 mW, 2 s light pulses for cell-body NpHR3.0 stimulation) that began when the mouse entered the light-paired chamber and continued for the entire duration that the mouse occupied the chamber. After 15 min, the side that was paired with photostimulation was swapped and mice were free to explore all three chambers for an additional 15 min reversal phase.

Three-chamber sociability assay

A three-chamber sociability assay was performed in an arena with three separate chambers as previously described¹⁷. On day one, the test mice were habituated to the arena, with two empty wire mesh cups placed in the two outer chambers for 5 min. On day two, the test mouse was placed in the center chamber and a conspecific juvenile (3–5-week-old males) was placed under one of the wire mesh cups. The test mice were first placed in the center chamber for 2 min. The barriers were then raised and the test mouse was allowed to explore freely for a 20-min session with 5-min epochs of photostimulation (0.4 Hz, 8 mW, 2 s light pulses for cell-body NpHR3.0 stimulation). The order of these epochs (LightOn or LightOff) was counterbalanced across mice. The placement of juvenile mice in the chamber was also counterbalanced across sessions.

Histology

Mice were transcardially perfused with 4% (w/v) paraformaldehyde (PFA) in phosphate-buffered saline (PBS) and postfixed in PFA overnight. Coronal sections 70 μm thick were immunostained for GFP (primary: chicken anti-GFP, Aves 608 #GFP-1020; secondary: goat anti-chicken 488, Invitrogen #A-11039), and/or mCherry (primary: rat anti-mCherry, Invitrogen #M11217; secondary: goat anti-rat 594, Invitrogen #11007). Primary antibodies were used at a concentration of 1:1000 and incubated on a shaker overnight at room temperature. Secondary antibodies were used at a concentration of 1:750 and incubated on a shaker for 2 hours at room temperature.

Statistics and reproducibility

Statistical analyses were performed in MATLAB (MathWorks) and Python. No statistical methods were used to predetermine sample sizes, which were based on previous experience with the variance of the assays. Statistical parameters such as the types of statistical tests, sample numbers (n) and statistical significance are reported in the figures and figure legends. T-tests were used to test if a sample mean was different from zero. Wilcoxon rank-sum tests were performed to compare unpaired measurements between two groups, and Wilcoxon signed-rank tests were conducted to compare paired measurements. For multi-group comparisons, we conducted one- or two-way analyses of variance followed by Tukey’s honestly significant difference procedure. All statistical tests were two-sided. All error bars in the figures are s.e.m. unless otherwise noted.

Data and code availability

Data and code are available from the lead contact upon reasonable request.