Cognitive skills and the evolution of social systems

ABSTRACT

How do animal social skills influence evolution? Complex animal social behaviors require many cognitive skills including individual recognition and observational learning. For social systems to evolve, these abilities need to be transmitted genetically or culturally and supported by the evolution of underlying neural systems. Because animal skill sets are so varied, it seems best to describe animal cognitive behaviors as being a social calculus that can change with experience, which has evolved to match and facilitate the complexity of the social system where it arose. That is, acquiring and using social information in response to a rapidly changing complex world leads to social competence enabling success in essential behavioral interactions. Here, we describe the remarkable suite of social skills discovered in the African cichlid fish Astatotilapia burtoni, including an attention hierarchy, male deception, transitive inference, the mechanistic bases of social dominance, female mate choice and the neural control of female reproductive behavior. The social calculus of this species is presented as an example of a potential causal factor in the evolution of sophisticated social behavior necessary for the evolutionary success of their social system.

Summary: Animals interacting successfully use cognitive skills such as recognizing individuals, their social rank and logic as described here in a cichlid fish, and the neural bases of these skills are identified.

Introduction

Behavior is the focal locus of selective pressures on animals as they forage, find habitats, find mates, produce offspring and navigate social relationships with conspecifics. Behavioral mechanisms mediate individual fitness and hence are the drivers of evolutionary change (e.g. Duckworth, 2009; Bateson and Gluckman, 2011). Characteristic species-specific behavioral patterns are favored by selection, leading to optimal behavioral rules (McNamara and Houston, 2009). Animal species typically first respond behaviorally to the challenges in their environments, followed by adaptations in morphology, physiology and life history, which typically change much more slowly (Relyea, 2001). Consequently, behavioral plasticity and flexibility are considered advantageous in natural contexts generally, allowing species to respond effectively to changing conditions (Wilson, 1978; Wright et al., 2010). While it is clear that behavior and behavioral responses to new challenges are key components of successful species, does social behavior, mediated by direct interaction or amongst conspecifics contribute to the evolution of species?

Generally, the remarkable complexity of animal social systems raises the question of how social interactions have shaped and been shaped by evolution. Ultimately, the success of complex organisms, shifting from solitary to complex lives, must have resulted from information transfer across generations, but what elements of social behavior might have contributed to the evolutionary success of species? It seems likely that the use of cognitive skills needed to navigate complex social situations such as individual recognition and observational learning may contribute to behavioral flexibility favoring selection. For social systems to evolve, these abilities need to be transmitted genetically or culturally and ultimately supported by the evolution of underlying neural systems.

That animals can exhibit complex socials skills raises the contentious issue of their intelligence. At one extreme, Descartes considered animals as complex machines like clocks without memories whose behavior is entirely triggered by internal or external stimuli (Seed et al., 2009). At the other extreme, Darwin proposed that the difference in minds across animals, including humans, ‘…is one of degree and not of kind’ (Darwin, 1882). Students of animal cognition recognize the tension between these two ends of a spectrum and understand that there must be an essential interplay between these disparate notions as neither captures the whole story (i.e. Dickinson and Shanks, 1995). Because animal skill sets are so varied, it seems best to describe animal cognitive behaviors as being a social calculus that can actually change with experience and has facilitated the complexity of the social system where it arose (Seyfarth and Cheney, 2003). This skill does not imply sophisticated intelligence as complex societies can be produced by simple mechanisms. That is, acquiring and using social information in response to a rapidly changing complex world leads to social competence, enabling success in essential behavioral interactions (e.g. Brosnan et al., 2010; Taborsky and Oliveira, 2012). It seems probable that each animal social system has evolved its own specialized social calculus, adapted to its senses, natural history and developmental trajectory. Here, I characterize the range and variety of social skills of an animal with a reasonably complex social system. The ultimate goal is to understand how individuals regulate complicated social interactions and how these might contribute to the evolution of social systems. I argue that behavioral flexibility in social behavior in the face of new challenges may contribute to social evolution, although these descriptions cannot establish their potential causal roles.

Dominance hierarchies and their role in social evolution

One ubiquitous organizing principle of social species is dominance hierarchies, which are the result of contests over limited resources such as food, mates, territories, etc. Social status may be established initially through physical contests but is typically maintained by specific social signals between individuals. The valence of social signals depends on the relative social status of the animals communicating and may result in a change in the hierarchy. As physical contests require a significant investment in time and energy and have a potential cost of physical damage (Huntingford and Turner, 1987), animal social systems have evolved to avoid these, usually signaling with threats or postures that typically emphasize body size and often teeth. Comparing two closely related cichlid fish species from Lake Tanganyika, Hick et al. (2014) found that highly social species were more likely to resolve disputes through submissive displays as compared with non-social species, suggesting that the dominance hierarchy formation is likely linked to the evolution of complex social systems.

Central to a successful social calculus for each individual in a social hierarchy is the ability to collect information from a variety of sources including prior experiences and ongoing observations. Animals can draw inferences from these observations to predict and anticipate the future behavior of others. Recognition of individuals is essential for many social behaviors, particularly in fluid social situations (e.g. Zulandt Schneider et al., 2001). For example, winning fights tends to increase the chances that the individual will continue to win in the future, and losing has the opposite effect (Hsu et al., 2006; Benelli et al., 2015a,b). In this context, observation or eavesdropping, which is the ability to monitor the behavior of other individuals, can be useful because this information can be used to reduce the need for future fighting (e.g. Grosenick et al., 2007).

Learning can be important for improving the chances an animal has for success. For example, improving fighting skills through observation has been shown to increase the chances of an animal becoming dominant because it can improve their fighting strategies through observation (Alcazar et al., 2014).

Model social system

To understand the mechanisms responsible for the evolution of social behavior, we can consider the cichlid fish species Astatotilapia burtoni (formerly Haplochromis burtoni), from Lake Tanganyika, East Africa. In this species, the male hierarchical social system requires a set of particular social skills to achieve and maintain high status. Astatotilapia burtoni is a unique species for understanding the role of social cognition in the evolution of social behavior because: (i) the social system is organized around resource guarding within small colonies of animals, a context that we can simulate well in a laboratory setting (Fernald and Hirata, 1977a,b; Fernald, 1977); (ii) male status is evident to observers because of their bright body color patterns and a lachrymal (eyebar) stripe, so behavior as a function of status is readily quantifiable (see Fig. 1); (iii) the neural systems responsible for social behavior include regulation of gonadotropin releasing hormone 1 (GnRH1)-containing neurons, which, in most species, ultimately controls reproduction – in A. burtoni, the size and connectivity of these neurons are regulated by male social status; (iv) levels of circulating hormones, tissue-specific peptides and DNA expression can be readily measured; and (v) the A. burtoni genome has been sequenced (Brawand et al., 2014), allowing us to measure gene expression in response to social situations (Desjardins et al., 2010), generate transgenic animals (Ma et al., 2015) and use CRISPR to delete key genes (Juntti et al., 2016).

Fig. 1.

Sketch of an observation site along the edge of a shore pool on the north end of Lake Tanganyika, near Bujumbura, Burundi, Africa. Solid dots represent grid stakes spaced at ∼50 cm intervals that label grids (1–5; A–D) for identification. Circles represent spawning pit locations of dominant Astatotilapia burtoni males. Lighter outlines circumscribe the approximate territories of dominant individuals. Male territories are located over the food source of detritus on the bottom of the pool. This detritus accumulates at the northeast edge of pools as a result of the strong daily southerly winds. Non-dominant males and females school together near the territorial area that they have to enter to eat. Based on Fernald and Hirata (1977a).

Social status of A. burtoni: the effect on reproduction

In their natural habitat, A. burtoni males exist as either dominant (D) or non-dominant (ND; see Fig. 1). D males actively defend territories and court females while ND males appear similar to the females that they mimic (Fernald and Hirata, 1977a). These two phenotypes are reversible depending on social circumstances. Importantly, D males are reproductively competent while ND males are not. The animals are located in colonies situated above food resources, where a limited fraction (10–30%) of D males occupy and defend territories (Fernald and Hirata, 1977a,b).

The evident D and ND male external phenotypic differences are reflected in major physiological and neural responses to differences in social status. As males switch from ND to D, expression of the black bar through the eye, brightening of the body color and switching of behavioral repertoires occur in minutes, while the physiological and neural changes caused by status change develop over several days. During social encounters, A. burtoni attend closely to the behavior of others, as judged from monitoring eye fixation points during social encounters (Fernald, 1985). The social environment is highly fluid and individuals must make quick decisions about what to do, depending on their perceptions of the immediate circumstances. Females observing contests between D males may be sizing up potential mates, as shown in mate-choice studies (Clement et al., 2005; see below), while ND males may be observing which D male they might defeat to acquire a territorial resource, as suggested by their behavior in experiments showing they can use transitive inference about the fighting abilities of animals they observe (Grosenick et al., 2007). There are often dramatic fights during which males engage in mouth-to-mouth biting, hitting each other with their bodies and nipping at each other's fins (Fernald, 1977). When a ND male challenges and successfully defeats a resident male, he rapidly turns on his bright body colors (Fernald and Hirata, 1977a; Burmeister et al., 2005), occupies the territory and begins performing the 17 distinct behaviors characteristic of a D male. During fights, it is clear that the animals use visual signals but the mechanosensory lateral line is also an important signaling system. In A. burtoni, the lateral line system is used during aggressive social interactions, as shown by comparing fighting in animals with and without a functional lateral line (Butler and Maruska, 2015). These mechanosensory signals may be used for non-contact assessment as well as during fights, possibly as a protective mechanism against physical damage.

A few days after ascent in the hierarchy (ND→D), the reproductive system of the ascending male is remodeled. The gonads grow and are populated with viable sperm, rendering the male reproductively competent as a result of changes at numerous locations in the hypothalamic-pituitary–gonadal (HPG) axis (Maruska and Fernald, 2014).

The brain controls reproduction in A. burtoni, as in all vertebrate species, by secreting GnRH1 from neurons in the hypothalamus. These neurons deliver the GnRH1 decapeptide to the pituitary, where it activates release of intermediate signals, follicular hormone and leutinizing hormone, which, in turn, cause the release of sex steroids from the gonads. What is unusual in this teleost fish is that these GnRH1-containing neurons in D males increase dramatically in size (Davis and Fernald, 1990), grow their dendrites (Fernald, 2012) and quickly increase production of GnRH1 mRNA (Burmeister et al., 2005) and GnRH1 peptide (White et al., 2002). Importantly, when the animal is socially dominant, the GnRH1 neurons are connected via gap junctions that facilitate the necessary production of pulses of GnRH1 (Ma et al., 2015). But social status depends on social context and animal size. If a D male enters an area populated by larger animals (∼5% longer), he dramatically turns off his eyebar and dims his body colors. Over a longer time scale (ca. 3 days), the GnRH1 neurons in the hypothalamus of the brain of the newly ND male shrink in size, producing less GnRH1 mRNA and peptide (Davis and Fernald, 1990; Francis et al., 1993; Fig. 2). In the same time frame, steroid hormone (e.g. androgen and estrogen) mRNA levels drop, as do GnRH receptor mRNA levels (Au et al., 2006; Burmeister et al., 2007; Harbott et al., 2007). In addition, certain key electrical properties of GnRH1 neurons change (Greenwood and Fernald, 2004).

Fig. 2.

Three-dimensional plot of mean soma diameter of preoptic area irGnRH1 neurons in males of different social status. There are significant differences between dominant (D) and D→non-dominant (ND) males and between ND and ND→D males. Soma size is a proxy for gonadotropin releasing hormone 1 (GnRH1) abundance. The percentage of individuals with mean soma size in a given size bin is plotted for each treatment condition. Redrawn from Francis et al. (1993).

Attention hierarchy in male A. burtoni

Attention, which is the selective concentration on some particular aspect of current conditions, presumably evolved to allocate limited neural resources to highly salient stimuli. Studies of attention represent a significant research effort among psychologists seeking the sensory cues and signals that generate attention in relation to other behavioral and cognitive processes as well as its neural bases. Chance and Larsen (1976), studying primates, first suggested that their social hierarchy is a structure of social attention, with higher ranking individuals receiving more attention than lower ranking animals. They referred to subordinate animals attending very closely to the behavior of dominant animals as an ‘attention hierarchy’ (Chance and Larsen, 1976). Subsequent research with a variety of primate species found that organization within complex primate social systems depended on relatedness, age, etc. and social interchange. As a consequence, status is communicated with a combination of social gestures, postures and vocalizations in addition to attending visually to others (see Rowell and Olson, 1983; Johnson and Karin-D'Arcy, 2006). In A. burtoni, however, relatedness and age do not appear to play a role in the status hierarchy, which is quite fluid and depends on the ongoing aggressive encounters. In general, however, it is clear that animals in social groups monitor the behavior of conspecifics and use such observations to guide their ongoing behavior. For example, animals may attend to male fighting, females choosing mates or many other social interactions. And males watch higher ranking animals carefully, presumably to calibrate potential opportunities for social ascent.

Rank and attention hierarchies figure in the behavior of groups in many ways. For example, children in a school setting adjust their behavior as a function of their status relative to other members of the group (Boulton and Smith, 1990). This is manifest in the reaction evoked when a high-ranking individual attacks or threatens a lower ranking individual. As is also seen in animals, the attacked animal often then attacks an individual of still lower rank (Vaughn and Waters, 1981). Even in animals with less-complex social connections and limited means of interaction, such as many reptile species (Summers et al., 2005), visual attention hierarchies may play an important role in communication.

Astatotilapia burtoni turn their bodies frequently during aggressive encounters and can orient towards another animal at up to 1900 deg s⁻¹ through a wide range of angles (Fernald, 1975). Most interestingly, when D males swim slowly towards ND males, the ND males actually move away from their positions, evidently anticipating a likely attack by the D male (Fernald, 1985; Fig. 3). This suggests that A. burtoni attend to one another through a visual attention hierarchy. Desjardins et al. (2012) hypothesized that ND males attend to D males, possibly even anticipating their movements. They video recorded aggressive encounters between D and ND males in an aquarium that had only one territory centered on a shelter occupied by a D male, and discovered that D and ND males were never aggressive simultaneously. Surprisingly, the largest of the ND males would transiently behave aggressively and even court females when the D male could not see him because the D male was in his shelter (Fig. 4). However, when the D male returned to action, he chased and attacked the ND male and females, though he did not target specific individuals.

Fig. 3.

An example of eye movements during social interactions. Fish and their eye positions have been drawn from individual film images at 160 ms intervals with the 1st to 11th frames labeled. Relative eye locations of the dominant animal are shown by lines extending from stalks attached to the eyes at an arbitrary angle behind the central axis of the eye. Note that the ND animals move out of the region being approached by the D male well before he arrives. Redrawn from Fernald (1985).

Fig. 4.

Schematic illustration of typical D male behavior in the presence of an intermediate ND male attempting to ascend to dominance. Large rectangles represent the D male has entered his shelter and cannot be seen by the ND males. Thin black bars show when an individual chases or attacks another fish. Note that intermediate ND males only attack other animals when the D male is in his shelter and cannot see them. From data presented in Desjardins et al. (2012).

ND males transiently express aggression and courtship when out of view of the D male, behavior that never occurs when the D male is present. In particular, Fig. 4 shows evidence that the ND males on some occasions anticipate the D male swimming into his shelter and begin to act like a D male in advance of the actual D male entering his pot. Overall, it is evident that the ND males are exploiting information they collect visually about the D male to regulate their behavior, an example of the social calculus used by ND males.

Male deception

Deception in animals is often taken as a hallmark of a level of cognitive skills typical of primates and related to more complex kinds of social interactions. Clearly, this kind of ability, when used to gain advantage over competitors, could potentially be very useful. To determine whether A. burtoni males deceive each other with regard to their social status, Chen and Fernald (2011) placed a large male and a small male fish with an appropriately sized female in an aquarium with a ‘shared’ terracotta pot shelter, divided into halves by a removable opaque and watertight transparent partition (Fig. 5). The general experimental goal was to see how a small D male responds to a larger D male if he can only see the larger fish when the opaque divider is removed.

Fig. 5.

Schematic diagram of aquarium used for behavioral observations. Front view of an aquarium (45 l), divided in half with a watertight, clear divider (gray mid-line) and a removable opaque barrier (black mid-line); in one side there is a small male fish (left) and in the other there is a large male fish (∼4 times larger, right). The half terracotta pot (red curved line) was cut so that the two fish ‘shared’ the same shelter, though they were not aware of each other's presence. This ‘shared’ shelter was hemisected by both center dividers. A layer of gravel covered the bottom of the tank. Once the two males had established their territories on the opposite sides of the barrier, the black divider was removed. Modified from Chen and Fernald (2011).

Initially, both animals behaved like normal D males, courting the female, moving gravel from their hemi-pot and showing other typical D male behaviors. On the third day, when the opaque barrier was lifted, although no physical or chemical contact was possible, the larger male made several ‘attacks’ on the small male by swimming rapidly towards the smaller male behind the barrier in a threatening posture. In response, the smaller male quickly lost his coloration, including his eyebar. The behavior of the smaller male was typical of an animal losing his territorial status, leaving the shelter and excavating a new pit at the opposite corner of the tank. The changes in the smaller male resulted entirely from visual information, which also produced a reduction in androgen hormone expression for the first 3 days after removal of the black barrier (Chen and Fernald, 2011). Seven days after the visual exposure began, the smaller male had normal hormone levels but retained the coloration of a ND male. Behaviorally, he was observed courting the female when she was not visible to the larger male. Visual suppression alone clearly caused dramatic changes in the aggressive and territorial behavior of the smaller male, but did not produce lasting physiological changes. Thus, actual attacks rather than visual threats are needed to suppress reproduction in ND males.

The smaller male presented a false outward appearance in these conditions, not congruent with his internal physiology. This apparent deceptive behavior permitted him to continue his courtship despite the ongoing visual presence of the larger male. Thus, the smaller male recognized that the clear barrier prevented physical attacks by the larger male, although the smaller male attended to the position and orientation of the larger male, typically turning his tail towards the threatening larger male, consistent with an attention hierarchy as described above.

Transitive inference by males

ND males are continuously trying to ascend to dominance to allow them to become reproductively competent (Fernald, 1977). But how do ND males determine whether to attempt the ascent? In their natural habitat of Lake Tanganyika, East Africa, colonies of A. burtoni may range in size from a few dozen animals to over 100, depending on the area of the feeding substrate (Fernald and Hirata, 1977a). ND males could, in principle, fight each D male in a colony to gain a territory. However, the possibility of fighting with tens of animals to find a territory holder weak enough to beat would obviously be prohibitive and dangerous. If the ND males could recognize successful or non-successful outcomes of aggressive encounters, they could choose which D male to challenge. As described above, the attention hierarchy evident during social encounters suggested A. burtoni might have observational skills that would allow them to predict the outcome of male–male encounters. That is, males could infer their chances of winning a fight simply from watching pairwise fights of other animals. We (Grosenick et al., 2007) wanted to know whether males watching the outcomes of other males fighting could predict fight outcomes. Specifically, could a male observing an encounter where fish A beats fish B, and fish B beats fish C, infer that fish A could beat fish C using logical inference from these observations?

Transitive inference is a form of deductive reasoning that allows inferring of a relationship among items that have not been explicitly compared. To test for the ability of such logical skills, typically the individual is taught some comparisons. For example if A is longer than B and B is longer than C, transitive inference is the ability to reason that A is longer than C. Piaget (1928) described this as a key milestone in the development of human infants older than ∼3 years and it has also been described for non-human primates (Gillian, 1981; McGonigle and Chalmers, 1977; Rapp et al., 1996), rats (Davis, 1992; Roberts and Phelps, 1994) and birds (Bond et al., 2003; Steirn et al., 1995; von Fersen et al., 1991; Weiss et al., 2010).

To discover whether A. burtoni had the ability to infer fighting ability from watching selected fish fight, Grosenick et al. (2007) arranged for ‘bystander’ fish to watch paired fights and from them infer a hierarchy amongst the fighters. Pairwise fights were staged between five size- and color-matched animals (a–e) by moving one fish into another fish's tank, where the intruder lost, so that a>b, b>c, c>d and d>e, an implied hierarchy of a>b>c>d>e (Fig. 6). Control animals fought without an implied hierarchy (i.e. a=b=c=d=e).

Fig. 6.

Tank arrangement and bystander training. Five rival males (a–e) were arranged in visually, chemically and physically isolated compartments around a central bystander unit. To train a bystander on a particular fight, the male scheduled to be the ‘loser’ was removed from his unit and placed in the territory of the scheduled ‘winner’. The opaque barrier separating the bystander from the rivals was then removed to allow the bystander to view the fight. Fish were trained for either a>b>c>d>e or a=b=c=d=e. The fight d versus e (e ‘wins’, d ‘loses’) is shown here in diagrammatic form. Modified from Grosenick et al., (2007).

Grosenick et al. (2007) then monitored the bystander fish for their choice of the possible winner in a fight between animals they had not previously seen together. They found that the bystander fish moved towards the weaker animal (Oliveira et al., 1998; Clement et al., 2005; Fig. 7); for example, when viewing fish b and d, the bystander chose d as being weaker. For transitive inference to be possible, the fish must be able to recognize individuals, which these fish obviously did.

Fig. 7.

Arena for testing male choice after observing fights. Following training in the reconfigured test arena (A), fish were tested in a novel arena (B). Fish moved towards the weaker animal (d) (Oliveria et al., 1998; Clement et al., 2005), indicating they considered b would beat d in a fight. Modified from Grosenick et al. (2007).

Why would transitive inference be useful for A. burtoni? In their native habitat, which is in temporary shore-pools and estuaries, environmental factors (hippopotamuses, wind, predation, etc.) disrupt territories, so keying on the features of their prospective opponents independent of context could be valuable (Fernald and Hirata, 1977a,b). Moreover, as only D males can reproduce, males continually attempt to rise in social status through fighting. By limiting their fights to ones they estimate they will win, ND males increase their chances of reproductive success. Ascent from ND to D is rapid and activates many processes we have identified (Burmeister et al., 2005; Maruska and Fernald, 2014). Transitive inference may be important for many group-living animals facing constraints on reproduction.

The experiments described above relied entirely on visual information being provided to the individual, although chemosensory information is also essential for many animal species including fish. Maruska and Fernald (2012) tested whether A. burtoni uses other sensory channels in addition to vision and the lateral line when communicating with conspecifics. They injected dye into D males and tested whether these individuals used urine pulses as a part of their sensory signaling. D males increased their urination along with territorial behaviors when they were visually exposed to another male (Fig. 8). This study of contextual chemosensory urine signaling shows that urine signals very likely play a complementary role to visual signaling.

Fig. 8.

Pulsatile urine release from a dominant male. A D male, injected with dye was exposed visually to another D male. The D male produced pulses of urine in response to the other male (arrow). After Maruska and Fernald (2012).

What internal factors might control dominance?

Social status hierarchies are widespread among animals (Wilson, 1975), but little is known about the cellular and molecular mechanisms that might underlie them. Recent research on epigenetic mechanisms influenced by behavior led to the hypothesis that these rapid changes in gene expression patterns without altering the genome might play a role in the rapid and important changes in social status that occur in many species. There are many types of epigenetic change, including histone acetylation, histone methylation, chromatin modification and DNA methylation (Razin, 1998). One of the best-studied epigenetic mechanisms, DNA methylation, occurs when a methyl group is covalently added to a cytosine. Such changes are attractive because they can occur in minutes and are reversible, making them compatible with rapid behavioral changes (Metivier et al., 2008).

Reports suggest a role for DNA methylation in a variety of behavioral contexts, including caste in insects (Maleszka, 2008), learning and memory in vertebrates (Day and Sweatt, 2011), and a shift from hive work to foraging in honey bees (Herb et al., 2012).

Lenkov et al. (2015) tested the role of DNA methylation in the establishment of social status. They reared A. burtoni males so that they had never been dominant (NBD), as a result of suppression by larger conspecifics. When two size- and color-matched NBD males were placed in a tank that could only sustain one territory, fighting began immediately and in <<30 min, one male became D. This is consistent with both field (Fernald and Hirata, 1977b) and laboratory (Fernald, 1977) observations.

Using this paradigm, Lenkov et al. (2015) then experimentally altered the DNA methylation by injecting one NBD male with either a DNA methylating or a DNA de-methylating agent and the other with vehicle. Remarkably, those animals with chemically increased methylation were highly likely to ascend in rank and those with inhibited methylation processes were highly unlikely to ascend to the D rank (Fig. 9).

Fig. 9.

Experimental design and results from testing injections. (A) Males that had never be socially dominant (N=60) were randomly divided into three groups of size-matched pairs. (B) In control animals, both members of the pair received vehicle injections. In experimental animals of one group (left), one member of the pair received l-methionine and the other received vehicle control; in the other group (right), one member of the pair received zebularine and the other received vehicle control. (C) Animals receiving l-methionine injections became socially dominant while those receiving zebularine did not. Modified from Lenkov et al. (2015).

Thus, DNA methylation state plays a key role in establishing social dominance, presumably through changes in methylation levels of some suite of genes or genomic regions, biasing the animal’s chance of becoming dominant. How could this be the case? For A. burtoni to be successful in a social context, it must act appropriately for its social status. I suggest that some epigenetic process may serve as a temporary mark or memory of social status. Ongoing measurements of the genetic substrate should reveal the actual genomic mechanisms responsible.

In other species, DNA methylation has been shown to influence social rank. Kucharski et al. (2008) showed that silencing an enzyme (DNMT3) responsible for transferring methyl groups to DNA in bee larvae produced bees with queen-like characteristics. In addition, Herb et al. (2012) showed that the division of labor in a bee colony can be reversed through DNA methylation marks in ganglia.

In one other case, rhesus macaques (Macaca mulatta), social rank has been shown to regulate aspects of the immune system (Tung et al., 2012). Specifically, low-status individuals increase immune response gene expression to inflammation. The immune response was mediated through the stress response producing methylation state changes of 694, or ∼70%, of the rank-related genes between high- and low-ranking animals. DNA methylation may be linked to social behavior and specifically to social status in animals.

Genomic responses in females choosing mates

How does the female brain respond to social information relevant to a choice between potential mates? Information about mates can change female behavior dramatically, which makes sense given that choosing the right mate is extremely important for the survival of her offspring. What cognitive activity might accompany behavioral and physiological changes in females responding to possible mates with different attributes? Specifically, how does the female brain respond to social information about mates?

To discover which neurons in the female brain are activated by social stimuli from males, Desjardins et al. (2010) analyzed expression of immediate early genes (IEGs) in the brain regions that make up the vertebrate social behavior network. IEGs are transcription factors that comprise the first wave of gene expression induced as neurons are activated. Extensive experimental work has shown that a range of natural experiences, including sensory stimuli, can induce IEG expression and, consequently, it has been used extensively in mammals and birds (e.g. Mello and Clayton, 1994; Rusak et al., 1990) to identify neural systems engaged in social responses. In A. burtoni, Burmeister and Fernald (2005) showed that egr-1 is highly conserved and that it responds robustly ca. 30 min after stimulation. Similarly, c-fos is also a valuable genetic signal for detecting brain responses in A. burtoni (Burmeister and Fernald, 2005).

We predicted that activity in the highly conserved vertebrate social behavior network would be an indicator of socially important information. The social behavior network (SBN), first described by Newman (1999), is a suite of brain nuclei whose activity has been mapped relative to numerous social behaviors such as male and female sexual behavior, aggressive behavior and parental behavior. SBN neuroanatomical homologs have been identified in fish and birds (Goodson, 2005; Goodson and Bass, 2002). Although these brain nuclei are well known to respond to behavioral actions, it was not known whether they might also respond to social information. Clement et al. (2005) showed that reproductively ready (i.e. gravid) females associated preferentially with D males, while non-gravid females prefer ND males.

Desjardins et al. (2010) tested females by placing them in an aquarium with size- and color-matched D males at each end, separated by a clear watertight Plexiglas barrier. Females could see but not physically interact with the males and preferences were based on measuring her proximity to a particular male over 20 min. After the female chose, she was shown a fight between the two males arranged by moving one male into the other's compartment. Control females chose between two males but did not subsequently see a fight. The hypothesis was that distinctly different patterns of IEG expression would be generated depending on whether the females saw their chosen male win or lose a fight. To test this, gene expression patterns of cfos and egr-1 were compared in six brain nuclei that are part of the SBN, using RT-PCR (Desjardins et al., 2010).

The IEG expression patterns were found to be strikingly different. Specifically, females who saw a preferred male win had increased activation in brain nuclei that are tied to reproduction and reproductive behavior, including the anterior hypothalamus, ventromedial hypothalamus, preoptic area and periaqueductal gray. In contrast, females who saw a preferred male lose had increased IEG expression in the lateral septum, an area associated with anxiety (Desjardins et al., 2010).

Remarkably, female brain responses reflected activation in response to only visual information as the females had no direct interactions with the males in the experiment. Moreover, they clearly recognized and remembered specific individuals. Thus, social information alone caused activation of key brain areas, although the IEG expression is likely only a fraction of total brain activation. This suggests that social information is available to females to guide social decisions.

How does this information inform female mate choice? Desjardins et al. (2010) repeated the experiment but, after the female chose a male and saw the two candidates fight, she had to choose again. In the second choice, if she had previously seen her chosen male lose, she switched. In contrast, if her chosen male had won, she rarely switched her choice, which makes sense as females choose more dominant males.

Neural control of female reproductive behavior

What neural circuits mediate female reproductive behavior once a mate is chosen? With our understanding of some of the complexities of reproductive behavior, we have begun to dissect the neural circuits that control sexual behavior.

Fertile females that select a mate perform a stereotyped spawning routine, offering quantifiable behavioral outputs of neural circuits. We know in all vertebrates that the key signaling molecules rise with fertility to prime the brain for reproductive activity (e.g. Beach, 1976; McCarthy et al., 1986). Juntti et al. (2016) showed that, within minutes of prostaglandin F_2α (PGF_2α) injection, female A. burtoni showed a naturalistic pattern of sexual behavior, as had been shown in other teleosts previously (Stacey, 1976; Cole and Stacey, 1984; Villars et al., 1985; Liley and Tan, 1985; Kidd et al., 2013) (Fig. 10). Juntti et al. (2016) identified the cells in the preoptic areas of the brain that transduce the prostaglandin signal to trigger mating and showed that the gonadal steroid 17α,20β-dihydroxyprogesterone (DHP) modulates mRNA levels of the putative receptor for PGF_2α (Ptgfr). Doudna and Charpentier (2014) and Jao et al. (2013), using the CRISPR/Cas9 method, targeted a gene mutation in A. burtoni to show that Ptgfr is necessary and sufficient for the initiation of sexual behavior, uncoupling sexual behavior from reproductive status. This suggests that PGF_2α communicates fertility status via Ptgfr to circuits in the brain that drive female sexual behavior. In mammals, PGF_2α promotes both the onset of labor and maternal behavior, suggesting that PGF_2α signaling has a conserved ancestral function, causing the release of offspring or eggs from the reproductive tract.

Fig. 10.

Behavior and corresponding regulatory signals leading to female behavior. (A) Schematic illustration of the natural progression of spawning behavior. The male quivers his body in front of the female and leads her to his spawning site (flower pot) and they circle, with the male pecking the female to elicit egg laying followed by egg collection, and the female collecting the sperm from the male anal fin to fertilize the eggs in her mouth. (B) Neural pathway from the receipt of social signals triggering activation of GnRH1 neurons; release of luteinizing hormone (LH) and follicle stimulating hormone (FSH) stimulates the reproductive tract, where progestin (PR) and PGF_2α act on the Ptgfr receptor to stimulate the final stages of egg laying. Modified from Juntti et al. (2016).

Summary and conclusions

Living in obligate social groups with dynamic social interactions favors the evolution of social behavior, requiring enhanced cognitive abilities. Numerous studies have sought to identify and compare social cognitive skills in animals (e.g. Shettleworth, 2009). Studies on fish species can provide insights into cognitive skills because of the similarities in brain structure, with well-recognized homologies across all vertebrates (e.g. Goodson and Bass, 2002; Goodson, 2005; O'Connell and Hofmann, 2012). A recent review (Bshary et al., 2014) summarizes many recent findings about fish social cognition.

Dominance hierarchies are widespread among animals and are often referred to as pecking orders after their first description in chickens by Schjelderup-Ebbe (see Perrin, 1955). In A. burtoni, there are in essence local ‘pecking orders’ within a colony. Above, I have described for A. burtoni several interrelated social skills that have facilitated social success in the highly competitive cichlid social interactions. This model system offers unique opportunities because its natural behavior is well described and those conditions can be readily replicated in the lab. Moreover, the focus of individuals is reduced to the essentials of finding food and sex, with each member of the species having particular skills in these domains. The hierarchical rank system of males provides an excellent window on the important decisions and behaviors needed in order to gain dominance and hence be able to reproduce.

The remarkably rapid phenotypic changes in males caused by changes in status lead to physiological transformation of males. As A. burtoni males become dominant, making reproduction possible, a set of neural responses are triggered, including changing the size of important cells in the brain, and their connections, as well as many other processes (Maruska and Fernald, 2014). Clearly, the deceptive ND males can uncouple these processes. A behavioral transformation in response to a change in social status is typical in many species in which there are D and ND male phenotypes (Wilson, 1975). Important cognitive skills support the social calculus that enables males to assess and exploit their social rank.

Attention hierarchy

Individuals in a social group attend to the behaviors of others, as first described in primate social systems (Chance, 1976) and evident in humans as well. Observing others prepares the individual for social encounters and in particular for future interactions. Such an attention hierarchy is maintained in male A. burtoni and dominant animals receive more attention than others, which is considered to be the basis of rank. Lower ranking animals exploit this attention structure of the group by rapidly changing behavior when not being watched. This behavior likely serves to facilitate their ultimate social ascent. Such observer behavior has been seen in cleaner fish that are more cooperative in the presence of an audience (Pinto et al., 2011) and in predator assessment in stickleback fish (Webster and Laland, 2013).

Male deception

Males can deceive other males by acting ND when visually threatened but maintaining a readiness to mate. This surprisingly sophisticated kind of deception has not been widely reported among animals, though is known in birds concealing nests, but could be an important phenomenon. It allows a male to mate in the presence of dominant threats and requires a sophisticated phenotypic plasticity. Such a rapid adaptation to a novel situation might be the beginning of the evolution of a new trait similar to that of sneaker males.

Transitive inference

This social skill is the root of all logical behavior and is well described in primates and some bird species (see Grosenick et al., 2007). The discovery that A. burtoni can perform transitive inference shows that they recognize individuals and use that knowledge to their advantage. The mechanisms underlying transitive inference behavior in animals have been discussed extensively (reviewed in Grosenick et al., 2007). The debate is focused on whether transitive inference is an example of cognitive or associative learning. As there was no direct reinforcement of individuals in the paradigm of Grosenick et al. (2007), this must be considered cognitive learning; that is, the observers do not interact with the demonstrators, so a more complex representation must be present. The search for a neural representation of transitive inference is feasible in this species using IEGs or CRISPR.

Neural mechanisms responsible for key social skills

If complex social behaviors are essential for the survival of these animals, the goal is to understand how they are instantiated by the nervous system. Epigenetic modification of the methylation patterns in D males plays a significant role in dominance hierarchy. This discovery is consistent with findings in other species, including Macaque monkeys (Tung et al., 2012), and may provide a temporary mark of dominance. Finding the genomic responses to female mate choice, which identified key brain nuclei involved in important social decisions, and, similarly, identifying the key receptor controlling egg release in females are steps towards understanding how the brain controls complex behavioral patterns.

Possible roles of social skills for the evolution of social behavior

The fish behaviors described here have evolved in response to the requirements of a dynamic social system. Such social behaviors, like other behaviors, can play a causal role in evolution. How social behavior shifts as a result of learning or selection, and alters selective pressures in the short term to ultimately impact the rate of evolutionary diversity, remains an open question. The complexity of the social calculus used in response to changing conditions could provide a substrate for the evolution of new social behaviors, just as morphological traits are known to do. Individuals that more successfully navigate their social landscape and respond flexibly to novel situations will be more likely to find mates and territories, leading to improved reproductive success. Discovering the cellular and molecular substrates for the social skills, as described here, should lead to a better understanding about how the genome and nervous system can be transformed.