The role of extragroup encounters in a Neotropical, cooperative breeding primate, the common marmoset: a field playback experiment

Abstract

In cooperatively breeding species, encounters with intruders may serve multiple functions ranging from reaffirming group territory ranges to facilitating assessments for additional breeding opportunities. While these distinctive events offer the opportunity to investigate the delicate balance of these social dimensions within animal societies, their unpredictable occurrence makes witnessing and controlling these events in the wild particularly challenging. Here we used a field playback approach to simulate conspecific territorial incursions in cooperatively breeding common marmosets (Callithrix jacchus) to distinguish between the three following non-mutually exclusive functions of intergroup encounters in this species of New World primate: territorial defense, mate defense, and assessment of breeding opportunities. For these experiments, we systematically broadcast species-typical long-distance contact calls – phees – commonly used in intergroup interactions from the core and periphery of the groups’ territories using either male or female vocalizations. Consistent with a territorial defense hypothesis, a group’s reaction was independent of the simulated intruder’s sex and the response strength was greater when the playback stimulus was broadcast from the core areas of groups’ territories relative to stimulus broadcast from periphery areas. However, sex differences in some facets of their responses suggest that this is not the only potential function for these encounters. Mated males and females started to move first in response to simulated intruders of the opposite sex, suggesting that these events offered opportunities to assess extra-pair breeding opportunities, while the occurrence of females’ piloerection towards simulated female intruders is suggestive of mate-guarding. These data provide unique experimental evidence for the theory that excursions by conspecific intruders may serve multiple functions in a cooperatively breeding vertebrate and are reflective of the known complexities of common marmoset sociobiology.

Despite substantial variability in the organization of social groups, ranging from large fission-fusion organizations (e.g. African elephants - Archie et al., 2006; spotted hyenas - Smith et al., 2008; spider monkeys and chimpanzees - Symington, 1990) to smaller groups comprised of pair-bonded individuals and their offspring (e.g. prairie vole - Carter et al., 1995; titi monkeys - Bicca-Marques & Heymann, 2013; songbirds - de Kort et al., 2009), territoriality is a common behavior pattern amongst vertebrates (McGregor 1993; Clutton-Brock, 2016). Yet, despite the spatial segregation of social groups, encounters with neighbors and transient conspecifics are relatively common (Kinnaird, 1992; Sillero-Zubiri et al., 1996; Young et al., 2007) and can involve behaviors ranging from affiliative to contact aggression (Majolo et al., 2005; Kitchen & Beehner, 2007; Nichols et al., 2015).

While many studies aim to test why and when intergroup aggression occurs (Kinnaird, 1992; Fashing, 2001; Cooper et al., 2004; Kitchen et al., 2004; Korstjens et al., 2005), fewer data are available to address the significance of affiliative behaviors during encounters with outgroup conspecifics (Zhao, 1997; Majolo et al., 2005; Nichols et al., 2015). Potential explanations for intergroup aggression are related to food resource and mate defense (Kinnaird, 1992; Heinsohn & Packer, 1995; Bee & Gerhardt, 2001; Fashing, 2001; Cooper et al., 2004; Kitchen et al., 2004; Matthews, 2009), while explanations of intergroup affiliative behaviors are biased toward mating and dispersal opportunity assessment (Wiley, 1973; Taborsky, 1994; Temeles, 1994; Majolo et al., 2005; Nichols et al., 2015). In fact, a single encounter between groups could serve each of these functions, given that group members do not necessarily act cohesively during these events and behaviors with distinct functional significance are displayed by different individuals simultaneously (Fashing, 2001; Cant et al., 2002; Hale et al., 2003). For instance, intergroup encounters in Tana River crested mangabey can involve behaviors which vary from sexually presenting towards extra-group individuals to herding of sexually receptive females of the same group, indicating the significance of these encounters for mate defense and the opportunity for extra group copulation (Kinnaird, 1992). In more extreme cases, such as the observed for banded mongoose, affiliative behaviors such as extra-group copulations can take place even during violent encounters with resulting injuries and death (Nichols et al., 2015).

Dissecting the complexity of these encounters presents notable logistical challenges – particularly with respect to transient intruders – because of the difficulties in witnessing these events. Experimental techniques in the field, such as playbacks, offer opportunities to effectively simulate the presence of intruders and to directly examine the respective behavior of each individual in the group (McComb et al., 1994; Bee et al., 1999; Mennill et al., 2002; Illes & Yunes-Jimenez, 2009; Caselli et al., 2015). Here we sought to utilize field playbacks to simulate territorial incursions by conspecifics in common marmosets (Callithrix jacchus) in order to test the potential function significance of these pivotal social events for this Neotropical cooperatively breeding primate.

Common marmosets offer unique opportunities to examine the relative impact of multiple social pressures on individuals’ behavior during extra-group interactions. These small primates form cohesive groups of 3 to 15 individuals; including two or more adults, their offspring, and even unrelated individuals (Schiel & Souto, 2017). As a result, both breeding adults and animals that have reached sexual maturity - but are not yet actively producing offspring - are present in the group and contribute to caring for the young (Digby & Barreto, 1993; Schiel & Souto, 2017). The cooperative nature of their society extends to several facets of their social cognition (Schiel & Huber, 2006; Miller et al., 2016; Miller, 2017) and, as a result, the species has been argued to exhibit prosocial tendencies commonly associated with humans (Burkart et al., 2009; Burkart & van Schaik, 2010). However, this affiliative dimension of common marmoset society seems restricted to group members, as they commonly show aggressive displays toward potential intruders and neighboring groups (Hubrecht, 1985; Stevenson & Rylands, 1988; Lazaro-Perea, 2001). Despite the aversion to outsiders, evidence suggests that extra group copulations are not uncommon in this species (Digby, 1999; Lazaro-Perea, 2001). Therefore, encounters may serve multiple functions by reaffirming groups’ identity and territory ranges while also allowing for mate defense and facilitating assessments for additional breeding opportunities, especially by nonbreeding individuals (Lazaro-Perea, 2001; Digby et al., 2006). Further exploration of intergroup interaction offers the opportunity to effectively investigate the delicate balance of these social dimensions in common marmosets.

Interactions with extra-group individuals typically take place at the periphery of a group’s home range and commonly involve all group members (Lazaro-Perea, 2001). Because of the species’ small body size and arboreal lifestyle, these encounters are commonly associated with vocal signals - such as the species-typical long-distance phee calls – that are emitted for communication between conspecifics (Hubrecht, 1985; Stevenson & Rylands, 1988; Bezerra & Souto, 2008). In fact, conspecific intruders will often announce their presence by producing phee calls (Hubrecht. 1985; Lazaro-Perea, 2001). Because this vocalization communicates critical social information about the caller - such as its individual identity, sex, and group dialect (Norcross et al., 1994; Miller et al., 2010; Miller & Thomas, 2012; Zurcher & Burkart, 2017) - listeners will be able to identify the caller as a territorial intruder and behave accordingly.

Given that encounters with individuals from outside the group may serve multiple distinct, but parallel roles in common marmoset sociobiology (Lazaro-Perea, 2001; Digby et al., 2007), we tested the functional importance of these distinctive social interactions in mate and territory defense as well as in the assessment of breeding opportunities. To test these non-mutually exclusive hypotheses, we performed a series of field playback experiments in which we simulated intruders by broadcasting phee calls produced by either unknown male or unknown female callers within the group’s core area and at the periphery of its territory. We initially predicted that phee calls produced by an unknown intruder should elicit distinctive patterns of behavior based on subjects’ sex and mating status. More specifically, if individuals outside the group primarily elicit a territorial defense response, we expect adults to react to simulated intruders independently of the caller’s sex. Likewise, a more robust behavioral response to playbacks broadcast from the core area of their home ranges than from the periphery would be expected, since intruders in the center are believed to pose a greater threat to the territory owners (Giraldeau & Ydenberg, 1987; Crofoot & Gilby, 2012). As an intruder can signal a breeding opportunity, phee calls could also elicit patterns of behavior in individuals that are the opposite sex of the simulated interloper - such as moving to the playback location more quickly - to assess that individual more closely. Likewise, a same sex intruder could also be perceived as a threat and elicit mate guarding behaviors in mated individuals, including a higher incident of agonistic displays and moving more quickly towards the intruder.

METHODS

Study Site

This study was conducted in the semiarid Caatinga scrublands at Baracuhy Biological Field Station (7°31’S, 36°17’W) in the municipality of Cabaceiras, state of Paraíba, in Northeastern Brazil. The study region is in one of the driest areas of Brazil. The area is characterized by a hot semiarid climate, receiving approximately 500 mm of rain per year and with temperatures reaching up to 40 °C. The rainy season lasts from February to July and the dry season from August to January. The vegetation is predominantly low, characterized by arboreal shrubs and scattered trees (see De la Fuente et al., 2014, for detailed information about the study site).

Subjects

The groups at the study site ranged from four to ten individuals and regularly engaged in vocal interactions with at least one neighboring group. Natural encounters are not frequent, occurring at rates of about 0.17 per day (S.C.N. Castro and P.H.B. Ayres, personal observation). From May through December 2016, we monitored the ranging pattern and conducted playback experiments with three fully habituated groups that were approximately 300 m apart from each other (Fig. 1). Given that phee calls cannot be transmitted efficiently beyond 100 m (Morrill et al., 2013), the selected groups had no visual or acoustic contact with each other. Group 1 was composed of three adults (two males and one female) as well as two infant males at the beginning of the study. The adult female and one infant disappeared and a new female came into the group in August 2016. At the end of the study, group 1 was composed of three adults (two males and one female) and one juvenile male. Group 2 was initially composed of five adults (four males and one female), two juvenile females, and two infant females. In June 2016, one adult male and one juvenile female disappeared. At the end of the study, group 2 was composed of four adults (three males and one female), one juvenile female, and two infant females. Throughout the entire study period, group 3 was composed of four adults (three males and one adult female) and two infants (one male and one female). We defined mated individuals as marmosets which were seen copulating during the observational period that took place prior to the playback trials, while we monitored the use of space by the groups to determine its ranges.

Figure 1.

Spatial representations of groups’ home ranges, significant resources and playback locations. (A) Maps on the left indicate the location of the study site, while the map on the right indicates the location shows an expanded view to indicate the location of each groups’ range at the site. Territorial ranges for Groups 1, 2 and 3 are shown. (B) Maps depict the significant Resources in the territorial range for Groups 1 (left), 2 (middle) and 3 (right). Feeding trees are indicated by black dots while sleeping trees are shown with white dots. (C) Maps depict the location of the playback broadcasts within the territorial ranges of Groups 1, 2 and 3. White dots indicate playback locations in the Core, while black dots indicate playback locations in the Periphery. (B & C) The polygons represent the total area calculated with Minimum Convex Polygon (MCP) using 100% of location points (group 1(A): 11.6 ha, group 2(B): 5.26 ha, group 3(C): 2.11 ha). The gray scale in the bottom graphics represents the utilization distribution (UD) estimated using the adaptive kernel method. The UD grayscale indicates the probability of finding the group in each location, with more frequently used areas in darker colors. The black circles highlight the location of core areas.

The animals were marked with colored collars for individual recognition (see Encarnación et al., 1990, Bicca-Marques & Garber, 2004) in a previous behavioral study conducted on site. The procedures involved in the capture and marking of animals complied with current Brazilian laws and adhered to the ASAB/ABS Guidelines for the Use of Animals in Research and American Society of Primatologists (ASP) Principles for the Ethical Treatment of Non-Human Primates. This study was also approved by the governmental System of Authorization and Information on Biodiversity - SISBIO (No. 46770–1) and by the Ethics Committee on Animal Use (CEUA) of the Federal Rural University of Pernambuco (131/2016).

Ranging Pattern

To identify the areas with potential higher (Core) and lower (Periphery) value within groups’ home ranges, we monitored the three groups from dawn to dusk for at least 15 days over a period of three months (Group 1: from May through July, plus three extra full days in September, totaling eighteen days; Group 2: from May through July; Group 3: from September through November). Using a GPS receiver (Garmin eTrex Legend^® HCx), we recorded groups’ locations every ten minutes as well as the location of important resources, such as sleeping trees and important feeding sites (trees or shrubs used for fruit and gum consumption for at least two scan samples, or 20 min, in one day or used on consecutive days). We plotted the total area used by each group using a Minimum Convex Polygon (MCP; Hayne, 1949) with 100% of location points. To describe the intensity of range use, we estimated the utilization distributions (UDs) using the adaptive kernel method implemented with the KernelUD function of the “adehabitatHR” package (Calenge, 2006) of R software version 3.2.5 (R Development Core Team, 2016) with the default method for the estimation of the smoothing parameter (the ad hoc method). A UD gives the probability of relocating the groups at places within its range (Powell, 2000). We identified the groups’ ‘Core’ areas by locating the portions within groups’ territories that combined more intensely used areas and concentrated important resources (sleeping trees and important feeding sites). The ‘Periphery’ consisted of the remaining portions of groups’ home range outside the core areas (Fig. 1).

Stimulus Recording and Preparation for Playback Experiment

Since all phee calls used as test stimuli were recorded from individuals housed at the UCSD Cortical Systems and Behavior Laboratory (La Jolla, CA, USA), subjects in the field had no prior experience with these callers. Phee calls were recorded from six adult males and six adult females using standardized procedures (following Miller & Wang, 2006). Two individuals were placed approximately 3m apart on opposite ends of a sound attenuated chamber. A cloth occluder was placed equidistant from the individuals at the center of the room. A directional microphone (Sennheiser ME66) was placed in front of each subject and all vocalizations were recorded directly to disk. Phee calls were selected as stimuli based on high signal-noise ratio and absence of background sounds.

To evaluate whether subjects’ response was due to conspecific stimuli and not in reaction to any broadcasted sound, we also tested groups’ reaction to recordings of stripe-backed antbird (Myrmorchilus strigilatus) as a control stimulus. This species is common at our study site (BirdLife International, 2017) and its vocalizations do not seem to disturb or call the attention of common marmosets (S.C.N. Castro and P.H.B. Ayres, personal observation). The recordings of stripe-backed antbird were provided by the Macaulay Library of Cornell Lab of Ornithology (http://macaulaylibrary.org).

Experimental Design and Presentation

We used three different types of playback stimuli sets - male phees, female phees, and control stimuli. Each stimulus set comprised a series of 4 exemplars broadcasted with a 15s inter-stimulus interval. For the phee call stimuli, the calls of only a single animal were used within a given stimulus set, but 12 different callers (6 male/6 female) were used over the course of the experiment. We broadcasted these stimuli at two locations within group’s home range – in ‘Core’ area and at the ‘Periphery’ of the range. Each study group was presented with all playback stimulus types - male phees, female phees, and control stimuli - at both locations for a total of six individual playback trials for each study group (18 trials in total across all three groups). To avoid pseudo-replication, phees produced by each UCSD common marmoset were played only once (following Wiley, 2003). In other words, the stimuli of different callers were used for each test group. We conducted only one trial per day and randomized the order of treatments assigned to each group.

We conducted playback experiments between 6am and 12pm from September to December 2016. To simulate the invasion by conspecifics, we broadcasted the stimuli from inside groups’ range and within 25–30m of a group’s current location. The stimuli were presented using an Anchor MiniVox loudspeaker (Anchor, Carlsbad, CA; frequency response range: 100–15000 Hz, output power: 30 W, and maximum SPL: 109 dB) connected to an iPod Nano (Apple Computer Inc., Cupertino, CA). The loudspeaker was positioned at 2m from the forest floor to simulate realistic positioning of the animals while calling. All stimuli were normalized and the volume of broadcasting equipment was set to match the level of natural emissions produced by common marmosets, determined based on our field experience with natural emissions as well as pilot tests conducted prior to the experiment. Once established, we held this volume constant across all trials.

We began each trial when all the adults were in the sight of the observer, while foraging or resting, and only after a 30-min interval with no emission of phee calls from neighboring or focal groups. During each trial, one observer hidden behind vegetation broadcasted the stimulus while another monitored the subjects’ reactions for 30 min following the start of the trial.

Response Measures

During each playback trial, we recorded three categorical variables: [1] the identity of all individuals who reacted to the stimuli, [2] the first individual to exhibit an observable response (look towards the playback location or start to move), and [3] the occurrence of agonistic displays (piloerection). Piloerection is a commonly observed agonistic behavior in natural intergroup interactions (Hubrecht, 1985; Lazaro-Perea, 2001). Furthermore, we recorded five quantitative variables to determine the response intensity of the groups to the stimuli: [1] latency to move or to produce a vocal response after initiation of the playback stimulus, [2] percentage of monkeys in each group that started to travel towards the loudspeaker as well as [3] percentage arrived at loudspeaker location, [4] the speed of travel (distance travelled/time to arrive at loudspeaker location), and [5] the time spent in a radius of 5m from loudspeaker location (measured as the time from when the first adult entered the radius until the moment that the last adult moved outside of it). We calculated the percentage of individuals in each group that moved towards and reached the playback location to avoid the influence of groups’ size on the number of individuals that traveled towards the speaker. To avoid empty cells for the analyses, we assumed that the latency to move was equal to the duration of the trial (30 min) and the remaining quantitative variables were scored as zero whenever the groups did not behave accordingly (e.g.: when no monkey reached the playback location).

Statistical Analysis

We used a Generalized Linear Mixed Model (GLMM) to test the predictions regarding the subjects’ response strength to different stimuli sex and location, including stimulus type and location of playbacks as the fixed effects (explanatory variables) and the identity of groups as a random effect. To determine the significance of the models, we first compared the simplest models (with only one fixed variable) to the null model (including only the intercept and random variable). When the models with only one fixed variable accounted for enough variance to reject the null hypothesis, we compared the simplest models with the complete model, including the interactions between fixed effects (Stimuli sex and location), to test for further improvement in the explained variance.

To determine whether the frequency of individuals’ reaction type (piloerection displays, start to move and reach the speaker location first) was sex-dependent, we used Contingency Tables (2 × 2) comparing the frequency of each reaction manifested by individuals of different sex and mated status (mated males and females, and unmated males; there were no unmated adult females in the groups) according to the conspecific stimulus types (male and female phee) and location of the playback broadcasts (Core or Periphery). The values expected by chance were calculated considering the total number of mated individuals as well as unmated males in groups during the time of the experiment.

Because groups did not exhibit any overt behaviors in response to the control stimulus, there was no variance in the monkeys’ response to it. As a result, we did not include the control stimulus in our analysis and focused on the responses of animals to conspecific stimuli. All analyses were implemented in R software version 3.2.5 (R Development Core Team, 2016). To fit the Generalized Linear Mixed Models, we used the packages “lme4” version 3.1–125 (Bates et al., 2015) and to perform model comparisons we used the anova function (likelihood ratio test) of “stats” package version 3.2.5. The significance level was set to 5% and the data are presented as X ± SD.

RESULTS

We first analyzed the salience of conspecific phee calls relative to control stimuli to determine whether detection of a conspecific intruder would elicit a response beyond what would be expected of any sound in the local habitat. While playback of stripe-backed antbird calls (control stimuli; N = 6 trials) elicited no response from marmosets, all playback trials in which a conspecific phee call was broadcast elicited a robust behavioral response (N = 12). In response to the phee calls of simulated intruders, most group members (percentage of group members: 78.3 ± 25.3) quickly started to move towards the position of the loudspeaker upon hearing the phee playback (latency time: 4.2 ± 4.8 s after initiation of the playback). Only a single individual – a non-mated adult male - emitted phee calls in 3 of the 12 conspecific trials (all in response to male stimuli: two broadcasts from the Periphery and one from the Core area). Although most members of groups started to travel towards the loudspeaker (speed: 17.0 ± 25.9 min), only a small percentage of group members actually arrived at the loudspeaker location (percentage of group members: 31.8 ± 41.4). Once arriving at the speaker location, the individuals remained within 5m of this area for roughly 12.8 ± 13.7 min.

Territorial Defense Hypothesis

Groups’ response, based on continuous variables, provided some support for the territorial defense hypothesis. We observed that marmosets’ response strength to playbacks were independent of the sex of the simulated intruder (Table 1; Fig. 2), but varied with the speaker location (Core vs. Periphery). Specifically, the rate of travel towards the playback location, the percentage of group members that arrived at playback location, and the time spent within the 5m radius of the loudspeaker was greater when playbacks were broadcasted from groups’ Core areas, relative to the Periphery. The latency to initiate travel and the percentage of group members that started to move were also independent of stimulus type (Table 1).

Table 1.

Result of model comparisons among null models and the models including single fixed effects (stimuli sex and loudspeaker location) as well as the comparison between the significant model with single fixed effect and the complete model, including the interaction among predictors variables.

Dependent variable	Model	X 2	df	P
Latency to move (min)	Null, Model 1 (Sex)	1.73	1	0.19
	Null, Model 2 (Site)	2.83	1	0.09
	Null, Complete model (Sex*Site)	5.54	3	0.14
Time interval in the radius of loudspeaker (min)	Null, Model 1 (Sex)	0.67	1	0.41
	Null, Model 2 (Site)	9.03	1	<0.01
	Model 2, Complete model (Sex*Site)	2.67	2	0.26
Percentage of group members that starts to move	Null, Model 1 (Sex)	0.03	1	0.85
	Null, Model 2 (Site)	1.37	1	0.24
	Null, Complete model (Sex*Site)	1.78	3	0.62
Speed to arrive in the 5m radius of the loudspeaker (m/min)	Null, Model 1 (Sex)	0.002	1	0.96
	Null, Model 2 (Site)	5.39	1	<0.05
	Model 2, Complete model (Sex*Site)	0.03	2	0.98
Percentage of group members that arrived at loudspeaker location	Null, Model 1 (Sex)	0.007	1	0.93
	Null, Model 2 (Site)	10.66	1	<0.01
	Model 2, Complete model (Sex*Site)	0.31	2	0.86

Figure 2.

Comparison of group’s reaction to simulated intruders in the Core and Periphery of each group’s home range. (A) Percentage of monkeys that arrived at loudspeaker location. (B) The speed to arrive in loudspeaker location. (C) The time spent in loudspeaker location). The horizontal line shows the median, the bottom and top of the box show the first and third quartiles, respectively, and vertical dashed lines show 1.5 times the interquartile range of the data (approximately 2 standard deviations). The dots beyond the vertical bars represent the outliers.

Breeding Opportunities Assessment and Mate Defense Hypotheses

To test whether intruders might be perceived as a positive (breeding opportunity) or negative (mate defense) reproductive event, we examined sex differences in categorical responses to the playbacks. Results provide somewhat of a mixed view. Both the mated male and female were more likely to move first in response to the calls of opposite-sex intruders in the groups’ Core area than would be expected by chance (Contingency Tables: mated female: X₁ = 10.9, P < 0.005; mated male: X₁ = 9.20, P < 0.005; Fig. 3). When phee calls of female intruders were presented in the core area of the groups’ range, mated females displayed more piloerection than expected by chance (Contingency Table: X₁ = 7.60, P < 0.01; Fig. 4), providing support for the mate defense hypothesis. The piloerection display of both mated and non-mated males, however, was independent of the intruder’s sex (Contingency Tables: mated male: X₁ = 3.40, P > 0.05; non-mated males: X₁ = 3.10, P > 0.05).

Figure 3.

Individuals which started to travel towards the loudspeaker. Mosaic plot depicting the association between stimuli sex and speaker location with individuals’ status (Mf: mated female; Mm: mated males; nMm: nonmated males). The width of each cell with respect to its axis indicates the proportional contribution of each variable level to the total. The colors represent the level of the residual (Pearson residuals) for each combination of levels, with the blue color indicating the cells in which the individuals of a specific status started to travel more often than would be expected by chance.

Figure 4.

Piloerection display by individuals’ status to playback stimuli presented from Core and Periphery areas. The mosaic plot depicts the association between stimuli sex and loudspeaker location with individuals’ status (Mf: mated female; Mm: mated males; nMm: nonmated males). The width of each cell with respect to its axis indicates the proportional contribution of each variable level to the total. The colors represent the level of the residual (Pearson residuals) for each combination of levels, with the blue color indicating the cells in which there are more observations of piloerection than would be expected by chance. The bar with circles indicate the trials in which the adults did not reacted with piloerection. While all female stimuli broadcasted from groups center stimulated piloerection in the mated females, it did not stimulate piloerection in males.

Notably, not all behaviors were consistent with these hypothesized functions. The frequency of arrival at the loudspeaker location when playbacks broadcasted from the Core and Periphery was independent of the sex of the simulated intruder (Contingency Tables: mated female: X₁ = 1.70, P > 0.1; mated male: X₁ = 3.40, P > 0.05; non-mated males: X₁ = 2.09, P>0.05)

DISCUSSION

Here we examined the response of wild common marmosets to simulated territorial intruders using field playbacks. These experiments were designed to test different hypotheses regarding the functional importance that extra-group encounters may play in the sociobiology of this cooperatively breeding New World primate. Overall, phee calls from unknown conspecifics broadcasted within the territorial range of three groups elicited consistent and robust behavioral responses. Individuals from all groups typically responded by rapidly moving towards the loudspeaker location and producing visual or – in a handful of occasions – vocal displays. This response contrasted with the lack of a response to our control stimulus (calls of the local stripe-backed antbird). Different aspects of groups’ and individuals’ reaction provide support for at least one aspect of all tested hypotheses, suggesting that extra-group encounters may play more than one meaningful role in the social lives of these monkeys.

The territorial defense hypothesis, for instance, is supported by evidence - based on continuous variables – that groups’ reaction was independent of the simulated intruder’s sex and the response strength was greater when the playback stimulus was broadcast from the Core areas of the groups’ range relative to trials in which the stimulus was preseented from the Periphery. For instance, a greater percentage of group members reached the 5m radius of the speaker, moved more rapidly towards the speaker, and stayed longer around the playback locations in the Core relative to the Periphery. The pattern observed suggests that common marmosets perceived intruder calls from within their Core home range as more significant than calls at the Periphery, being consistent with predictions from the territorial defense hypothesis. The relative position of the stimulus on the territory has been shown to influence the strength or nature of an individual’s response in previous studies on birds, canids, and primates, with increasing response strength towards the central areas of the territories (Stoddard et al., 1991; Molles & Vehrencamp, 2001; Darden & Dabelsteen, 2008; Crofoot & Gilby, 2012) since it is presumably the most valuable area within an animal’s home range (Giraldeau & Ydenberg, 1987).

The responses of common marmosets to the field playbacks, based on categorical variables, suggest that territorial defense is not the only motivation driving marmoset behavior in response to intruders, as these events also afford opportunities to assess new individuals for potential extra-pair mating. We observed that mated males and females started to move first in response to playbacks of simulated intruders of the opposite sex, lending support for this hypothesis in our data. From a female’s perspective, potential benefits of extra copulation include the opportunity to increase the quality of the father (through sperm competition; Clutton-Brock, 2016) or increase the genetic variability within litters (Møller, 1992), even in Callitrichids, given that the twins can be sired by different males (Díaz-Muñoz, 2011). Thus, it is not surprising that female’s infidelity is commonly observed in some cooperative breeding birds and mammals (Mulder et al. 1994; Whittingham et al. 1997, Leclaire et al., 2013). Indeed, the involvement of breeding females in extra-group copulations has already been observed for common marmosets (Digby, 1999). Considering that neighboring groups can be, to some extent, composed of related individuals - potentially due to migration into neighboring groups or groups division (Nievergelt et al., 2000) - the presence of an entirely unknown male, as simulated here, may represent a unique opportunity to improve offspring genetic variability. Thus, the apparent motivation of mated females to move first towards simulated new male intruders may be a strategy to assess the potential for additional breeding opportunities.

From the mated male’s perspective, opportunities for extra-group copulation with an unknown female may represent a low-cost strategy to increase reproductive success (Digby, 1999; Clutton-Brock, 2016). Extra-group copulations are far more common in breeding male than breeding female marmosets (Hubrecht, 1985; Lazaro-Perea, 2001; Yamamoto et al., 2014). The presence of an unfamiliar female, as simulated here, may reduce the costs involved in searching for additional copulations, thereby creating a scenario that favored polygyny (Dunbar, 1995) and resulting in decreased reproductive potential of females, given the biological constraints imposed to them (Clutton-Brock, 2016). This is particularly true for social organizations in which females rely on male aid for infant care, such as is the case with Callitrichids (Garber, 2017). Because males cannot rear multiple females’ offspring, competition for a pair-bonded male may be intense among females (Clutton-Brock & Vincent, 1991; Ahnesjo et al., 2001). Indeed, the level of competition among females in marmosets is believed to be high (Garber, 1997; Arruda et al., 2005; Yamamoto et al., 2014). Therefore, while males should demonstrate interest for female intruders, females should treat these individuals with aggression (Dunbar, 1995). This expectation is consistent with the observed increase of piloerection displays by females towards the simulated female intruders. The same antagonistic behavior was notably infrequent in male marmosets in response to simulated male intruders. However, it is important to consider that during playback trials the females were probably not in estrus (no copulations or attempt of copulation were observed and, based on the time of birth of infants, they were probably already pregnant during the experiment). Because males are expected to be more aggressive when estrous females are present (Cooper et al., 2004; Kitchen et al., 2004, Majolo et al., 2005), this could be an alternative explanation for the lack of male-male agonistic behaviors.

The food defense hypothesis could also explain the observed agonistic behavior of females, given that the reproductive success of females is supposedly limited by access to food (Emlen & Oring, 1977). Thus, females are more likely to compete for these resources (Sterck et al., 1997). However, in species in which males provide parental care, female intrasexual competition is expected to increase and, therefore, females should repel rival females to avoid potential reduction of the direct benefits received from males (see Rosvall, 2011 for a review on intrasexual competition in females). Hence, considering that among Callitrichids the infant survival is correlated with the number of adult males in the groups (Koenig, 1995, Garber, 1997; Bales et al., 2000), the interpretation of females’ behavior as a mate defense strategy seems a more parsimonious scenario.

An alternative explanation to the observed sex-specific response of mated males and females, but the lack of sex-specific response by non-mated males, is that extra-group encounters do not actually have a role in the assessment of breeding opportunities. The reaction of mated individuals would be a strategy of mate defense through the reinforcement of their mated position within the partnership by preventing its mate of being usurped (Hall, 2004). Nonetheless, under this alternative scenario we should have observed opposite-sex directed piloerection displays. Since these were not observed, it suggests that the breeding opportunity assessment hypothesis is a more plausible explanation for mated individuals’ behavior. Genetic studies in cooperative species have indeed detected expressive rates of extra-group paternity in mammals (Goossens et al., 1998; Griffin et al., 2003) and birds (Whittingham et al., 1997; Durrant & Hughes, 2005).

The lack of a sex-specific response by non-mated males to the simulated intruders was, however, notably surprising. It is possible that helpers adopt other tactics for breeding opportunities. One strategy would be counter-calling on a daily basis during intergroup vocal interactions, as observed for subordinate pied babblers (Humphries et al., 2015). This counter-calling behavior is commonly witnessed at the study site. Another strategy would be to make sporadic incursions into neighbors’ ranges, a behavior that has been observed for common marmosets in the Atlantic forest (Lazaro-Perea, 2001). During these forays, the non-breeding helpers advertise their presence by producing phee calls and extra group copulations can occur (Hubrecht, 1985; Lazaro-Perea, 2001). Thus, these incursions may serve the dual function of providing opportunities not only for non-breeding males to copulate, but resident breeding males and females to mate with a genetically different individual as well. Helpers extraterritorial forays accompanied by extra group copulations were observed in other cooperatively breeding species (Legge & Cockburn, 2000; Young et al., 2007) suggesting that this may be a commonly employed strategy to create breeding opportunities.

Ecological constraints known to limit the dispersal success, such as environmental harshness and unpredictable conditions (Emlen, 1982), may have also influenced helpers’ behavior. The semiarid conditions at the study site may limit a male’s propensity to leave an established group to form a new one. Although the reduced opportunity to breed in natal groups may outweigh the costs of dispersal for common marmoset females, the chances of inheriting a breeding position in natal groups are expected to be higher for males (Yamamoto et al., 2014). Thus, for non-breeding males an effective strategy would simply be to stay in established groups and cooperate. Cooperation in territorial defense is one of the ways that helpers can collaborate with its natal group (Gaston, 1978; Koenig & Dickinson, 2004). Helpers’ cooperation in infant care and in territorial defense would signalize its quality; what can possibly result in direct benefits by obtaining a share of the current reproduction (Emlen, 1996) or by increasing its chance to inherit the breeding position in its own group (Price, 1990; Löttker et al., 2004; but see Tardif & Bales, 1997). In cooperatively breeding vertebrates, territorial inheritance can be an important benefit of philopatry (Buston, 2004). It is likely that our experimental design did not fully encapsulate all the social pressures faced by common marmosets and the strategies they employ to overcome these challenges. Additional experimental studies will be needed to more fully understand the functional significance of territorial incursions by common marmosets.

Overall the findings reported here based on a field playback approach are broadly consistent with previous observational studies, suggesting a complex mating pattern and social organization in the cooperatively breeding common marmoset (Digby, 1999; Lazaro-Perea, 2001; Yamamoto et al., 2014). This study, however, has yielded significant insights into the complex strategies employed by marmosets of different social categories for responding to territorial incursions from conspecifics. The complex social dynamic involved in interactions with outgroup individuals – which is more conspicuous during encounters – reveal that group members do not necessarily act cohesively due to different, and sometimes conflicting, intragroup interests. Overall, our results suggest that extra-group encounters serve multiple, non-mutually exclusive functions in a cooperatively breeding nonhuman primate species and provide powerful experimental evidence of distinct behavioral strategies that emerge based on the sex and putative breeding position of group members. The methods used in this study can be applied to other species in order to obtain comparative data to dissect the functional significance of intergroup aggressive and affiliative behaviors in group living birds and primates.