Coevolution of cognitive abilities and identity signals in individual recognition systems

Abstract

Social interactions are mediated by recognition systems, meaning that the cognitive abilities or phenotypic diversity that facilitate recognition may be common targets of social selection. Recognition occurs when a receiver compares the phenotypes produced by a sender with a template. Coevolution between sender and receiver traits has been empirically reported in multiple species and sensory modalities, though the dynamics and relative exaggeration of traits from senders versus receivers have received little attention. Here, we present a coevolutionary dynamic model that examines the conditions under which senders and receivers should invest effort in facilitating individual recognition. The model predicts coevolution of sender and receiver traits, with the equilibrium investment dependent on the relative costs of signal production versus cognition. In order for recognition to evolve, initial sender and receiver trait values must be above a threshold, suggesting that recognition requires some degree of pre-existing diversity and cognitive abilities. The analysis of selection gradients demonstrates that the strength of selection on sender signals and receiver cognition is strongest when the trait values are furthest from the optima. The model provides new insights into the expected strength and dynamics of selection during the origin and elaboration of individual recognition, an important feature of social cognition in many taxa.

This article is part of the theme issue ‘Signal detection theory in recognition systems: from evolving models to experimental tests’.

1. Introduction

Individual recognition allows animals to adaptively alter their behaviour and differentially allocate effort depending on with whom they interact [1–4]. In order for recognition to occur, senders must produce distinctive cues or signals that are then perceived and acted upon by receivers [5–8]. A major question, then, is how selection should jointly shape the interacting phenotypes of senders and receivers to facilitate individual recognition.

The still rapidly expanding literature on the evolution of organismal recognition systems has included not only empirical descriptions of how such systems operate [9–13] but also investigations of why they operate as they do, the latter drawing from the theoretical insights of engineering optimization (e.g. signal detection/acceptance threshold theory) and evolutionary game theory [2,14–17]. The tested theories typically focus on the evolutionary optima of thresholds that optimally balance rejection and acceptance errors [8,18–20]. However, relatively few theoretical studies have sought to predict the dynamics of the evolution of recognition systems and relate these predictions to the evolutionary dynamics (e.g. strength and timing of selection) underlying both recognition ability and the cues or signals mediating recognition. Given that recognition systems are the basis of social cognition [21], understanding the selective dynamics of recognition systems is likely to provide novel insights into the processes by which social cognition evolves.

With respect to individual recognition, the evolutionary dynamics of sender and receiver traits may provide insights into the processes shaping signal diversity and cognition, respectively. Previous models examining the evolution of sender phenotypes used for recognition have examined the extent of diversity expected within a population in relation to various scenarios dealing with competition or choice [2,14,15,22,23]. These models have not considered, however, the dynamic interplay of sender phenotypic diversity with receiver cognition. Similarly, there are many authors who have considered the importance of social cognition as a potential driver of cognitive evolution, typically ignoring the role of sender phenotypes in potentially facilitating recognition [24–26]. Verbal arguments have been put forth that the elaboration of sender phenotypes to facilitate recognition may influence the extent of investment in recognition-related cognition by receivers [27], though mathematical models examining the dynamic interplay between investment in sender diversity and receiver cognition have been lacking.

There is a wide diversity of contexts in which animals use individual recognition to navigate their social environment, which include competition, cooperation, care and mate choice, underscoring the importance of the evolution of individual recognition for a wide range of social phenomena. When animals compete, the recognition of individuals can prevent or reduce costs of suboptimal aggression levels towards individuals of varying fighting ability (‘dominance’ recognition) or of varying threat (e.g. ‘dear enemy’ recognition) [28,29]. In cooperative settings, recognition can assess the likelihood of returning the benefits of reciprocal cooperation (e.g. ‘cooperator’ recognition or ‘partner choice’) [16,30]. Care of offspring often involves costly investments and the recognition of parents and/or offspring can reduce costs of misappropriating efforts to an incorrect individual(s) [31,32]. Mate choice can also be facilitated by individual recognition where displays and mating choices are temporally separated [33].

We have developed a simple dynamical model of the coevolution of investment in discrimination ability, which is neurologically and temporally costly [34], and investment in individual distinctiveness in the context of individual recognition. We note that the cost of investing in identity information per se is different from investments in the production of the general signal. For example, pigments used in generating individually distinctive egg shell patterns in murres (Uria aalge) carry some costs, but differential patterning of pigments is not expected to be costly [35]. Similarly, the urinary proteins that signal scent mark identity in mice are metabolically costly to produce [36], though the genetic mechanisms generating individuality are likely not costly in themselves [37]. Even where identity information exists, costs of generating more apparent signals would still apply, such as the risk of detection by predators due to calling. We use this model to address a number of outstanding questions about the evolution of individual recognition: (i) What will be the relative investments in recognition by senders and receivers that are favoured by selection, and under what conditions will these investments be asymmetric? (ii) Does some degree of pre-existing diversity and/or discrimination ability facilitate the evolution of individual recognition? (iii) What are the expected dynamics of trait evolution during the course of elaboration? And, (iv) how does genetic relatedness influence the dynamics or equilibrium of investments in sender and receiver traits?

2. The basic model of coevolution of individual recognition

We develop a simple model of the coevolution of the costly investment d in discrimination ability of a receiver and costly investment v in exaggerating or amplifying pre-existing, individually distinctive traits of a sender (which corresponds to trait variation at the population level). The investments d and v could have any metric depending on the organism, but are multiplied by constants e and f, respectively, to obtain the fitness costs of those investments. The model we develop is applicable across a wide diversity of recognition systems. To highlight the generality of the model, table 1 details possible ways for organisms to increase their investment in discrimination and distinctiveness and possible costs associated with these investments. Increases in a receiver's d or a sender's v are assumed to increase the probability that the receiver will correctly classify a sender that is being assessed. As is true in individual recognition among conspecifics, individuals in the model act as both a sender and a receiver and possess a value for both d and v. The model assumes pairwise recognition, so we ask whether pairs of individuals each with their own values of d and v would recognize each other or not. When pairwise recognition occurs, a benefit is gained for both parties. For example, the recognition context could include the identification of the kinship, cooperative tendency, sex or dominance rank of a recipient. We further assume that a benefit b is gained if and only if both parties correctly identify each other. For example, where there is competition or conflict, recognition is thought to be beneficial because it reduces the likelihood of excessively costly aggression [52], the costs might be avoided only if the both parties correctly recognize each other's relative fighting ability. In the latter case, the benefit b could be thought of as the reduction of the cost inflicted by mutual aggression enabled by successful mutual recognition. In a cooperative scenario, the benefit b can be seen as the statistically expected enhancement of cooperative benefits received from a partner upon correct mutual recognition (e.g. if either partner does not recognize the other, they part without the possibility of the exchange of benefits).

Table 1.

Relating model variables to biological examples.

	description	example	references
investment in discrimination (d)
perceptual abilities	ability to perceive and discriminate among differences in signal	detecting cuckoo eggs	[38]
learning/memory	acquiring and retrieving knowledge of signals	offspring recognition	[39,40]
pattern recognition	comparison of a signal against an internal template	species-specific song preference in crickets	[41]
cost of discrimination (e)
energetic costs	energy allocated to developing and maintaining tissue to detect signal	high energetic cost of brain tissue	[34,42]
tissue allocation costs	tradeoff costs of using tissue for recognition and not another purpose	paper wasps with individual recognition abilities have decreased the size of olfactory bulbs	[43]
sensory processing costs	neurological cost to processing and interpreting signal	extracting relevant information in noisy environments	[44,45]
detection costs	energy allocated to detecting signal	active sensing modifying behaviour to reduce stimulus ambiguity	[46,47]
investment in distinctiveness (v)
information content	the amount of information conveyed by the signal	facial colour variation in guenon monkeys	[48]
efficacy	the strength and/or reliability of the signal	environmental changes in chemical signals in lizards	[49]
cost of distinctiveness (f)
manufacturing costs	energy expended to manufacture signal	mouse major urinary proteins (MUPs)	[36]
byproduct costs	costs associated with other receivers detecting the signal	predatory bats eavesdropping on frog calls	[50]
opportunity costs	tradeoff costs of using signal for recognition and not for another purpose	tradeoff between individual recognition cues and quality cues	[27]
recognition costs	costs associated with being recognized by others	punishment of cheaters	[51]

In particular, we assume that the probability p of successful mutual recognition for a focal individual exhibiting receiver investment d and sender investment v in a population in which everyone else exhibits d′ and v′ is given by the following equation:

$$p[(d,\hspace{0.17em}v),(d^{\mathrm{\prime }},\hspace{0.17em}{v}^{\mathrm{\prime }})]=\left(\frac{d}{d+a/v^{\mathrm{\prime }}}\right)\left(\frac{d^{\mathrm{\prime }}}{d^{\mathrm{\prime }}+a/v}\right),$$ 2.1

where a is a constant that measures the intrinsic difficulty of discrimination. As required by assumption, p (which ranges from 0 to 1.0) increases as d or d′ increases or v or v′ increases, supporting the intuition that recognition should be more likely with greater investments in sender and receiver traits. Similarly, as a decreases (i.e. the recognition task is easier), then p increases. We further assume that investment in discrimination d entails a cost ed, where e is a constant that measures the efficiency of discrimination—the resource costs required to discriminate among individuals (lower e means intrinsically more efficient discrimination). We also assume that investment in exaggerating cue diversity v entails a cost fv, where f is a constant that measures the efficiency of exaggeration—the resource costs for producing a diverse cue (lower f means intrinsically more efficient exaggeration of underlying cue diversity). Example costs related to sender and receiver traits are described in table 1. Greater efficiency as reflected by lower values of e or f reflects lower costs to the organism in elaborating either the sender or receiver traits in question. We might expect, for example, animals with larger brains and well-developed sensory systems to pay less additional cost for an improvement in discrimination ability compared with the same abilities in an organism starting with relatively less investment in sensory or processing abilities.

We assume that the focal individual's overall fitness payoff W is equal to a baseline y, plus the successful discrimination benefit b times the probability p that both parties correctly recognize each other, minus the costs of investment in discrimination ability and individual distinctiveness (cue diversity exaggeration), i.e.

$$W=y+bp[(d,\hspace{0.17em}v),(d^{\mathrm{\prime }},\hspace{0.17em}{v}^{\mathrm{\prime }})]-ed-fv.$$ 2.2

Assuming that d and v can evolve independently of each other to their respective optima, the evolutionarily stable strategy (ESS) values of d = d* and v = v* are obtained by solving the following equations:

$${\left.\frac{\mathrm{\partial }W}{\mathrm{\partial }d}\right|}_{d=d^{\ast },v=v^{\ast }}=0\hspace{0.17em},{\left.\frac{\mathrm{\partial }W}{\mathrm{\partial }v}\right|}_{d=d^{\ast },v=v^{\ast }}=0.$$ 2.3

The solutions d* and v* are seen to satisfy (see Maynard Smith [53]).

$$\frac{ab{d}^{\ast }{v}^{\ast 2}}{{(a+d^{\ast }{v}^{\ast })}^3}=e,\frac{ab{v}^{\ast }{d}^{\ast 2}}{{(a+d^{\ast }{v}^{\ast })}^3}=f.$$ 2.4

Equations (2.4) together imply that

$$\frac{\hspace{0.17em}f}{e}=\frac{d^{\ast }}{v^{\ast }},$$ 2.5

from which we can derive the prediction that the relative values of the equilibrium investments will vary inversely with their costs. For example, as is intuitive, the lower the cost rate for investing in discrimination ability versus individual distinctiveness, the higher the optimal investment should be in discrimination ability relative to the investment in individual distinctiveness.

Equation (2.5) implies that d* = (f/e)v*, which can be back-substituted into equations (2.4) to yield solution with separated optimal investments satisfying two equations:

$$\frac{abef{d}^{\ast 3}}{{(af+e{d}^{\ast 2})}^3}-1=0,\frac{abef{v}^{\ast 3}}{{(ae+f{v}^{\ast 2})}^3}-1=0.$$ 2.6

Joint solution of the equations in (2.6) yields six complicated polynomial roots, with up to two that are real. The lower real one corresponds to an unstable equilibrium and the larger real one corresponds to a stable equilibrium. An immediate consequence of both equations is that as b, the benefit of discrimination, increases both stable equilibrium investments v and d increase, as makes sense since increasing both investments should increase the chance of reaping the benefit of successful mutual recognition. (The latter is seen mathematically from the implicit function theorem by differentiating the left side equation in (2.6) with respect to e and f, respectively, in both cases yielding positive expressions.)

We are also interested in the coevolutionary dynamics for general d′ and v′, which are given by the selection gradients on d and v, S_d and S_v, respectively:

$${\left.S_d=\frac{\mathrm{\partial }W}{\mathrm{\partial }d}\right|}_{d=d^{\mathrm{\prime }},v=v^{\mathrm{\prime }}},\hspace{0.17em}{\left.\hspace{0.17em}{S}_v=\frac{\mathrm{\partial }W}{\mathrm{\partial }v}\right|}_{d=d^{\mathrm{\prime }},v=v^{\mathrm{\prime }}}.$$ 2.7

These selection gradients are given by

$$S_d=\hspace{0.17em}\frac{ab{d}^{\mathrm{\prime }}{v^{\mathrm{\prime }}}^2}{{(a+d^{\mathrm{\prime }}{v}^{\mathrm{\prime }})}^3}-e,S_v=\frac{ab{v}^{\mathrm{\prime }}{d^{\mathrm{\prime }}}^2}{{(a+d^{\mathrm{\prime }}{v}^{\mathrm{\prime }})}^3}-f.$$ 2.8

These are the selection gradients describing selection on a rare mutant d or v in a population in which everyone else adopts investments d′ and v′. Positive values of the selection gradient will result in future increases in the value of the investment by the population, negative values in decreases in the value of the investment, and zero values indicate selective equilibria, either stable or unstable. The analysis of the phase portraits of the coevolutionary dynamics generated by these selection gradients at different population values of d′ and v′ is illustrated in figures 1 and 2. The general analysis of these dynamics generates the following conclusions, where d* and v* now represent the non-trivial stable equilibrium.

Figure 1.

Coevolutionary dynamics for investment in discrimination ability d and investment in amplifying cues used in discrimination, v. Red arrows are eight particular directional selection gradients scattered evenly throughout the phase portrait; the stream flow-lines are the evolutionary trajectories starting at any given population values of d and v. The lengths of selection gradients measure vector-sum strengths of selection and consequently are proportional to the speed of joint changes in d and v. Blue dots are stable equilibria. (a) Coevolutionary phase portrait for discrimination parameter a = 0.80, benefit of discrimination b = 2, efficiency of discrimination e = 0.40 and efficiency of cue diversity amplification f = 0.40. The benefit of discrimination is too weak to give rise to and maintain a discrimination system and only the trivial equilibrium (d* = 0, v* = 0) is stable. (b) Increased benefit of discrimination: discrimination parameter a = 0.80, benefit of discrimination b = 8, efficiency of discrimination e = 0.40, efficiency of cue diversity amplification f = 0.40. A higher benefit of discrimination is sufficient to create a stable non-trivial equilibrium which is reached only if the beginning discrimination ability and/or cue diversity are sufficiently high. The strength of selection and the speed of progress to the equilibrium are higher farther from the equilibrium (in regions where selection moves the population towards the equilibrium) and for lower starting values of d and v. (Online version in colour.)

Figure 2.

(a) Coevolutionary dynamics for investment in discrimination ability d and investment in amplifying cues used in discrimination, v. Red arrows are eight particular directional selection gradients scattered evenly throughout the phase portrait; the stream flow-lines are the evolutionary trajectories starting at any given population values of d and v. The lengths of selection gradients measure vector-sum strengths of selection and consequently are proportional to the speed of joint changes in d and v. Values of discrimination parameters: difficulty of discrimination a = 0.80, benefit of discrimination b = 8. Blue dots are stable equilibria. (a) The efficiency of discrimination, e = 0.20, is less than the efficiency of cue diversity amplification, f = 0.40. (b) The efficiency of discrimination, e = 0.40, is greater than the efficiency of cue diversity amplification, f = 0.20. A higher efficiency of discrimination (lower e) moves the discrimination system to a stable equilibrium of higher discrimination ability and lower cue diversity exaggeration, and vice versa. The strength of selection and the speed of progress to the equilibrium are still higher farther from the equilibrium and for lower starting values of d* and v* in regions where selection moves the population towards the non-trivial equilibrium. (Online version in colour.)

3. Results of the simple coevolutionary dynamics model

(a). Relative investment in sender and receiver traits depends on efficiencies

As one might expect, the equilibrium discrimination system becomes more sophisticated or exaggerated (i.e. d* and v* become higher) as the benefit of discrimination b increases, as the efficiency of discrimination increases (lower e), and as the efficiency of exaggerating cue diversity v increases (lower f). The relative investment in discrimination versus cue diversity exaggeration should depend directly on their relative efficiencies (figure 2).

(b). Pre-existing abilities are needed for recognition to evolve

If the benefit of discrimination b is sufficiently low, then d* = 0 and v* = 0, i.e. there is only a trivial equilibrium and no non-trivial discrimination system is stably maintained (figure 1). Thus, there is a threshold value of b above which a discrimination system can evolve to a stable equilibrium.

If the benefit of discrimination b is sufficiently high (such that it overcomes costs associated with recognition), then a non-trivial stable equilibrium of d* and v* exists, but this equilibrium can be reached only if the starting values of d′ and/or v′ are sufficiently high; otherwise, d′ and v′ decay to the trivial equilibrium (figure 1). If the starting values of d′ and/or v′ are sufficiently high to avoid attraction to the trivial equilibrium, the non-trivial equilibrium will be rapidly approached (figure 1). How large the benefit or initial starting values of d and v must be in order to reach the non-trivial equilibrium depends on the values of the other parameters in the model. It is not the case, however, that any value greater than zero will lead to the non-trivial equilibrium, as an unstable equilibrium in the lower left hand corner of the phase portrait can push sufficiently low non-zero values back down to zero (see figure 2a as an example). Thus, at least a moderate degree of pre-existing diversity in sender phenotypes or discrimination abilities in receivers is needed for individual recognition to evolve.

(c). Selection is stronger when populations are farther from the equilibrium

The rate of approach to the non-trivial equilibrium is generally higher the farther the starting state is from that equilibrium, as indicated by the fact that the selection gradients are of larger magnitude the farther d′ and v′ are away from the equilibrium (figures 1 and 2). The strongest selection and fastest change occurs when the investments are increasing from low values during the population's progress towards the equilibrium (figures 1 and 2).

4. Extending the simple model to include genetic relatedness among individuals

We can extend the basic model to the case of recognition between relatives of genetic relatedness r. A convenient way to do this is to assume that the fitness of a rare focal mutant is given by the following equation:

$$\begin{array}{r}W=y+(1-r)bp[(d,\hspace{0.17em}v),(d^{\mathrm{\prime }},\hspace{0.17em}{v}^{\mathrm{\prime }})]+rbp[(d,\hspace{0.17em}v),(d,\hspace{0.17em}v)]-ed-fv.\end{array}$$ 4.1

That is, when the mutant is rare, it has a chance r of interacting with a partner that shares its same genetically specified investment in discrimination ability d and v and a chance (1 − r) of interacting with a partner that exhibits the population values of d′ and v′. We can then solve for possible ESSs using the method in (2.3). The new (complicated) expressions can each be differentiated with respect to relatedness r to determine how the ESS (non-trivial stable equilibrium) will change as relatedness changes. These derivatives are, respectively,

$$\frac{ab{d}^{\ast }{v}^{\ast 2}}{{(a+d^{\ast }{v}^{\ast })}^3},\hspace{0.17em}\frac{ab{v}^{\ast }{d}^{\ast 2}}{{(a+d^{\ast }{v}^{\ast })}^3}.$$ 4.2

Since both expressions are >0 at a non-trivial equilibrium, it follows from the implicit function theorem that increasing genetic relatedness r increases the stable investments in both discrimination ability and individual distinctiveness. The latter result makes sense because successful mutual recognition benefits both parties, so a focal individual is willing to increase its investments with a genetic relative because it now receives both a personal benefit and an indirect genetic benefit from successful mutual recognition.

5. Discussion

The predicted characteristics of the stable non-trivial equilibrium in our individual recognition model yield a number of biological predictions for existing individual recognition systems with mutual benefits for successful recognition: the most elaborate such systems should be those where (i) the benefits of mutual recognition are the greatest and (ii) the costs of investments in discrimination ability and individual distinctiveness are least. (iii) Increased relatedness among individuals that benefit from mutual recognition would lead to further exaggeration of traits. Additional dynamical models should be investigated for recognition contexts in which there are conflicts of interest (e.g. when discriminating individuals encounter not only neighbours or cooperators but also deceptive interlopers or defectors) and in which senders or receivers receive unequal benefits.

The coevolutionary model presented here makes a number of predictions regarding the expected patterns of investment in sender and receiver traits that can be tested with detailed studies of single species or using phylogenetic comparative analyses. Here, we highlight some of these predictions and point to current literature relevant to the model as well as systems that may be particularly fruitful for testing the coevolution of receiver cognition and sender identity signals.

• (i)Individual recognition systems should be most elaborated when the marginal benefits of accurate recognition are greatest. For example, in competitive contexts, we might expect greater investments in recognition when costs of misdirected aggression are higher, as might be the case for species with weaponry or otherwise costlier fights [54]. Similarly, the likelihood and costs of misdirecting parental care to the wrong individual can vary across species. For example, parent–offspring recognition systems are especially well developed in a variety of species nesting in high density colonies, where the risk of misdirected parental care is especially high, such as in colonial penguins [55], swallows [56], gulls [57], pinnipeds [39] and bats [58]. While there have been some studies highlighting variation in individual distinctiveness of eggs, plumage or vocalizations [59–61] in these groups, attempting to tie the patterns of evolution to the relative frequency and costs for misdirected parental care is a clear avenue for future research.
• (ii)Recognition systems should be most elaborated when the marginal costs of investing in discrimination ability or individual distinctiveness are smaller. This prediction has two major consequences. First, costs of sender and receiver traits will differ across clades. Most obviously, this relates to body size, which limits perceptual capabilities and the ability to produce detectable signals [27]. All else being equal, slight increases in sensory or neural tissue are likely to be less costly for larger species [34], as the same increase in complexity would represent a relatively small portion of total energy/mass budget. Second, we expect more sophisticated recognition systems in species that already possess substantial discrimination abilities or individually distinctive phenotypes. Discrimination abilities may be selected for in contexts other than individual recognition, such as mating, foraging or predator recognition [62,63]. The costs of modifying and elaborating the pre-existing sensory systems or neural circuits underlying these other forms of recognition should be lower than for species in which such sensory systems or circuitry are absent and must be built from scratch. Likewise, many species that lack individual recognition nevertheless have traits with individually distinctive features [64–67] that would facilitate the evolutionary origin of recognition in comparison with a species that lacks individual identity cues. As a corollary to prediction (ii), recognition systems with high costs for investing in discrimination ability and/or distinctiveness suggest a substantial benefit of recognition. For example, major urinary proteins (MUPs) are used for scent marking and as a signal of individual identity in house mice (Mus musculus domesticus) [37,68] and constitute upwards of 20% of transcripts produced in the liver in highly competitive individuals [36]. The extreme investment in individually distinctive scent marks in house mice suggests a large benefit of territory marking for territorial males in that species, likely because mate choice is partly determined by females monitoring individual scent marking abilities [69,70].
• (iii)A further prediction of our model is that there should be a positive correlation between discrimination ability and individual distinctiveness across species, as the investments in each trait will generally exhibit positive coevolution since the benefits of both traits are increased by higher benefits of recognition. The model does not predict equal investments in sender and receiver traits, but rather that both traits are likely to increase given a greater benefit of recognition. Furthermore, the relative investment in recognition ability versus individual distinctiveness should be inversely related to the ratio of their marginal costs. For example, relative investment in recognition ability should be highest in species having elaborate, previously established recognition systems and low levels of interindividual variability. Nevertheless, the model predicts that increased benefits of recognition will be associated with some degree of increase in sender and receiver traits. For example, humans are highly adept at recognizing other human faces and our faces show signatures of selection to advertize individual identity [71,72]. Comparative data from halictid bees provide evidence for increased investment in antennal sensilla density in social halictid bees relative to solitary species [73].

Our model also provides insight into the dynamics of the evolution of new recognition systems. We find that recognition systems should be hard to evolve unless the benefits of discrimination reach some threshold value, and then should evolve rapidly to a stable equilibrium, provided that the beginning values of d′ and v′ are sufficiently large. Thus, the evolution of novel recognition systems requires some degree of pre-existing discrimination and distinctiveness in a population. New recognition systems may be favoured to evolve in response to a temporary or permanent increase in the benefit of recognition or the relatedness of individuals, such as a change in environmental conditions or a population bottleneck. Similarly, changes in the initial values of d′ or v′ could also favour the evolution of a new recognition system. One possible scenario by which this could occur is through gene duplication. For example, cuticular hydrocarbon genes that convey chemical cues for nest-mate recognition in social insects, and opsin genes which are necessary for colour discrimination, evolved through repeated gene duplication events [74,75]. An interesting extension of these observations is that the type of signal (e.g. visual, chemical or acoustic) used by a new recognition system should favour the modality with the lowest cost of discrimination and distinctiveness (table 1). This would not be determined by solely the signal-to-noise ratio for a given modality but also its external costs such as how vulnerable that modality may be to eavesdropping predators.

Based on our model, we predict a difference in the evolutionary dynamics of recognition traits between recently evolved and established recognition systems. Our model furthers the understanding of the adaptive dynamics of sender–receiver coevolution in signalling systems by applying adaptive dynamics analyses to a new context–individual recognition [76,77]. Once at the equilibrium, selection should be weak and therefore relatively difficult to detect. By contrast, selection favouring recognition systems far from equilibrium, as when investment in discrimination ability and individual distinctiveness are initially low in the early stages of evolution, should be relatively strong and thus easier to detect. Indeed, greater rates of fitness gain when populations are further from the equilibrium may be a common feature of adaptation across a range of taxa and traits [78,79]. Perhaps counterintuitively, this would suggest that species from clades where recognition traits are otherwise minimal will show stronger evidence of selection on receiver cognition or sender identity signals compared with clades with ancestrally well-developed recognition abilities. For example, we would expect weak signatures of recent selection on recognition traits in species such primates, where ancestral sender and receiver states were likely relatively close to current optima [80,81]. By contrast, strong selection may be detectable in species starting from minimal individual recognition abilities. Intriguingly, these predictions appear supported by recent evidence on the evolutionary dynamics of loci likely involved in the individual recognition ability of co-nesting foundresses (nest-founding females) of the paper wasp Polistes fuscatus. Female P. fuscatus have diverse facial markings that they use to recognize other individuals [28]. Polistes fuscatus are cognitively specialized for learning and memory of conspecific facial images [82] and possess robust long-term memories for familiar individuals [83]. Mutual recognition in P. fuscatus is thought to be beneficial because it reduces aggression by mediating dominance interactions among co-foundresses [52] and facilitates tracking of individual contributions to nest building and egg laying [84]. A recent study investigating the evolutionary dynamics of individual recognition in P. fuscatus found evidence for recent strong selective sweeps on loci associated with cognition and vision [85]. By contrast, studies examining recent selection in species with older established recognition systems such as the great tit [86], parrot [87] or the great apes [88] have found relatively weak selection across loci associated with cognition. Similarly, attempts to link cognitive phenotypes to genetic loci in humans have identified hundreds of loci with a small phenotypic effect [89] consistent with an evolutionary process of weak selection across a large number of loci. These findings support our prediction that selection should be strong early in the evolution of a new recognition system but weak in established recognition systems that are presumably nearer to the equilibrium, similar to the expectation from Fisher's geometric model of adaptation [90,91]. Thus, signatures of selection on social cognition may be most prominent in species with relatively modest cognitive abilities that have resulted from recent selection for increased investments in recognition.

Data accessibility

This article has no additional data.

Authors' contributions

S.E.M., M.J.S. and H.K.R. designed and conceived of the study; H.K.R. analysed the model; S.E.M., M.J.S. and H.K.R. wrote and edited the paper.

Competing interests

We declare we have no competing interests.

Funding

This work was supported by NIH grant no. DP2-GM128202 (M.J.S.), NSF CAREER grant no. DEB-1750394 (M.J.S.) and NSF Postdoctoral Fellowship grant no. DBI-1711703 (S.E.M.).