Variation and the response to variation as a basis for successful cooperation

Abstract

In applying game theory to problems in biology, differences between individuals are often ignored. In particular, when analysing the evolution of cooperation it is often implicitly assumed that ignoring variation will produce predictions that approximate the solution when differences are included. This need not be true. As we demonstrate, differences are not innocuous noise, but can fundamentally change the nature of a game. Even small amounts of variability can stabilize cooperation by, for example, maintaining the need to deal with cheaters. Differences promote the need to learn about others in an interaction, leading to contingent behaviour that can reduce conflict, and to negotiated outcomes that may or may not be more cooperative than unconditional actions. Once there are mechanisms such as mutation and environmental influences that maintain variation within populations, whether cooperation evolves may depend on the variation in the cooperativeness trait. Variation means that it may be worth taking a chance that a partner is cooperative by being cooperative. When there are markets, so that individuals can break off interactions to seek a better partner, variation promotes choosiness and hence penalizes those uncooperative individuals, who are rejected. Variation promotes the need to monitor the previous behaviour of others, and once this social sensitivity exists, the need to maintain a good reputation can promote cooperation.

1. Introduction

Variability is a characteristic feature of virtually all biological phenomena. The composition of populations and communities can fluctuate dramatically over time, and the phenotypes of the member organisms are no less diverse. In particular, behaviour is notoriously variable. It is only in rather extreme circumstances, such as for the response to a powerful stimulus indicating an immediate and severe threat to an animal's life, that one observes relatively uniform behaviour. The reasons for this phenotypic and behavioural variation are manifold, including both different kinds of genetic variation in populations and different environmental influences on the phenotype, as well as the inherent randomness in the expression of behaviour and other phenotypic traits. Our aim here is to examine the broad implications of such variability for the evolution and maintenance of cooperation and, in particular, to argue for the great importance of variation and the response to variation for the evolution of cooperation.

The basic principle we explore is, first, that when potential cooperative partners vary in their states and strategies, it can be profitable for an individual to interact with a potential partner in order to determine whether continued interaction will be worthwhile. This can be compared with exploration as a component of resource exploitation. Second, the conditionality implicit in the individual's response to the information obtained about the potential partner can promote a more cooperative behaviour in the partner, and vice versa. In this way, variability acts as a fuel for the evolution of cooperation.

The errors made by players when selecting actions are important features of game theoretical analyses, where they are used as motivation for equilibrium or solution concepts, such as sub-game perfection and trembling-hand perfection (Binmore 2007). In game theory, one frequently investigates situations where such mistaken actions occur with low probability, or in the limit of the mistake probability going to zero, and this form of variability has played a fundamental role in the game theoretical study of cooperation. Nevertheless, our main point here is that variability has an even greater impact on the evolution of cooperation than traditionally assigned to it. The broader importance is related both to the magnitude of variability, which need not be small, and to its nature.

The kind of variation that is of relevance for the evolution of cooperation would certainly include phenomena that fall under the headings of animal and human personality variation (Dall et al. 2004; Sih et al. 2004; Soares et al. 2010), as exemplified by the analysis of trustworthiness by McNamara et al. (2009), but also other types of variation in the needs, abilities and other characteristics of potential cooperative partners. For instance, for the much discussed egg-trading procedure in the Black Hamlet fish (Fisher 1980) it has been suggested that individual variation in the availability of eggs implies that a failure to reciprocate an egg-laying bout is a reliable indicator of a lack of eggs in the partner, and thus provides information about partner quality (Leimar 1997a). Variation in some aspect of partner quality is in fact a basic prerequisite for the evolution of partner choice and the operation of any form of system resembling a market. That variation in partner quality can lead to the evolution of partner choice illustrates an important general theme: variation in a trait at the population level can select for individuals to develop other traits.

We will develop our topic by examining a number of related but separate approaches and examples. This includes a brief overview of the classical use of mistaken actions in game-theoretical analyses, the importance of variation for processes of negotiation, and ideas relating to partner choice and markets. Along with our treatment of theoretical concepts and models, we also emphasize that the perspective of variation acting as a fuel for cooperation can be an important point of contact between theory and observation. For instance, the hypothesis that cooperative behaviour entails exploration of, and response to, certain kinds of variation is very suited to empirical investigation.

2. Game theory and mistaken actions

Errors in the execution of a strategy are inevitable and should therefore realistically be included into game theory modelling. The original motivation for their inclusion was a desire to rule out certain Nash equilibria that rely on empty ‘threats’. These can occur in games with sequences of moves, where a player could dissuade a co-player from choosing a certain action through a ‘threat’ of some form of punishment. The threat is called empty if the player would do best by not choosing the ‘punishing’ action, were a ‘transgression’ actually to occur. The general idea used to avoid such Nash equilibria is to allow for a small probability of error, so that all positions of the game are guaranteed to be reached with a positive probability regardless of which strategies are played. Strategies involving ‘empty threats’ can then no longer be part of a Nash equilibrium.

To illustrate these ideas, consider two parents that have common young. Each parent must decide whether to care for the young or to desert. The young dies if no parent cares, but do as well if one parent cares as if both care. Thus, since care involves costs, each parent would do best if it deserted and the partner cared. Let us assume the male decides first. The female then makes her decision, basing what she does on the male's decision. Suppose that the male's strategy is to care and the female's strategy is to always desert, regardless of whether the male cares or deserts. Then these two strategies are in Nash equilibrium; since the male cares it is best for the female to desert, and since the female will always desert it is best for the male to care. This equilibrium is maintained by the female's threat to desert if the male deserts. However, since the male always cares the female never needs to carry out this threat. The situation is altered if the male makes the occasional mistake and deserts rather than cares. This seems biologically realistic if the male may be killed by a predator while out searching for food—he will then not return and has effectively deserted. Once the male has deserted, the female would do better to care than to carry out her threat and desert herself. Thus, the threat by the female will not evolve when males make errors.

Errors mean that there is a positive probability that the female will find herself in both of the two possible circumstances (male cares, male has deserted) when she makes her decision. There is then only one best strategy for her: desert if the male cares, care if the male has deserted. Faced with this strategy, the male does best to desert. The male and female strategies are then best responses to one another and are in Nash equilibrium. However, this second Nash equilibrium is now stable even if the male makes the occasional error. Note that at this equilibrium each individual makes the best decision under all possible circumstances. An equilibrium with this property is referred to as a sub-game perfect Nash equilibrium. In general, the existence of occasional errors leads to solution concepts, such as sub-game perfection and trembling-hand perfection (Binmore 2007). For games with perfect information, backward induction (that is working backwards from the last decisions) becomes a powerful method of finding such a solution. The point we wish to stress about these classical developments is that variation can have a qualitative influence on solutions to a game.

For evolutionary game theory, the possibility of errors in the execution of a strategy has had a strong influence on the analysis of important games. Early on there was a realization that a popular strategy such as ‘tit-for-tat’ for a repeated Prisoner's Dilemma game is in fact not an evolutionarily stable strategy (ESS; Selten & Hammerstein 1984), but that errors would allow qualitatively similar strategies to be evolutionarily stable (Boyd 1989). The ESSs for the repeated Prisoner's Dilemma game that emerged from these analyses, including ‘contrite tit-for-tat’ (Boyd 1989) and ‘win-stay, lose-shift’ (Nowak & Sigmund 1993), together with a great number of other strategies (Leimar 1997b), can be thought of as regulating behavioural noise in a cooperative relationship. If mistaken actions occur on a regular basis in relationships, strategies ought to be adapted to this social environment.

3. Beyond sub-game perfection

For games with a fixed finite number of rounds, sub-game perfection entails that players must choose optimally in the final round. Given this behaviour in the final round, players must chose optimally in the next-to-last round, and so on, following the logic of backward induction. If the component game in each round is a Prisoner's Dilemma, this reasoning arrives at the conclusion that defecting in all rounds is the only sub-game perfect Nash equilibrium. It is then striking that certain kinds of variation in strategies give rise to qualitatively different conclusions.

McNamara et al. (2004) considered a game in which population members meet pairwise to play a number of rounds of the Prisoner's Dilemma against each other. There are a fixed maximum number of rounds known to both players, but the game ends prematurely if either player defects on a round. The payoff to a player is the sum of payoffs over those rounds that are played. Since the game ends immediately following a defection, a strategy simply specifies the number of rounds to cooperate before defecting. For a certain range of parameters, it is easy to show that since it is better to defect before a partner does so, defection on the first round is the unique Nash equilibrium. This equilibrium is sub-game perfect. However, whether this strategy evolves in an evolutionary simulation depends on the amount of genetic variation generated by mutation.

When the mutation rate is low, most individuals are the same. It turns out that the best response is then to be less cooperative than the population mean, and consequently the Nash equilibrium strategy of defection on the first round evolves. In contrast, at higher mutation rates there is uncertainty as to the number of rounds for which the partner will cooperate. It is then worth taking a risk that the partner is more cooperative than the population mean. If after several rounds of play the partner has not defected, then it is probably that the partner is more cooperative than the population mean. It may thus be worth it to be more cooperative than the population mean, and consequently high levels of cooperation can evolve (figure 1).

Figure 1.

Evolutionary outcome for each of two mutation rates ɛ in the model of McNamara et al. (2004). The trait under selection is the number of rounds of cooperation in the Prisoner's Dilemma before defection. There are a maximum of 10 rounds. The figure shows the proportion of population members with each trait value. Circles with continuous line, ɛ = 0.017; triangles with dashed lines, ɛ = 0.018. (Reproduced with permission from McNamara et al. 2004.)

In this example, the direction of selection depends on the amount of variability in the trait. Variability in the population is maintained by mutation, but other sources of variability such as genetic recombination would have similar effects. When cooperation evolves individuals in the population are approximately, but not exactly, optimal given the population composition. However, this behaviour is very far from that at the sub-game perfect Nash equilibrium. It may thus be misleading to treat the variation as just a small amount of noise and to assume that the sub-game perfect Nash equilibrium will be a good approximation to the behaviour that evolves.

4. Partner quality variation

In cooperative exchanges, the size of a partner's contribution could depend on the partner's ability to supply help and also on the partner's need for help in return. As noted by Boyd (1992), in all likelihood there will be variation in the need for help and in the ability to provide help, both from one individual to the next and over time for a given individual. We would expect an individual to regulate cooperative contributions according to its needs and abilities and, as a consequence of the partner acting in the same way, to also regulate its contributions according to available information about the partner's characteristics (Leimar 1997a). Since the partner's actions provide information about partner quality, such regulation of a relationship tends to follow the principle of reciprocity. Receiving little or no help is an indication of low partner quality, making further investments in such a partner less profitable. These intuitive ideas have been explored and verified through game-theoretical analysis by Leimar (1997a), and there is additional work arriving at qualitatively similar conclusions (Lotem et al. 1999; Fishman et al. 2001; Sherratt & Roberts 2001). These effects of partner quality variation have also been incorporated into broad theories of the evolution of interspecific mutualisms (Foster & Wenseleers 2006).

Provided that a cooperative relationship is regulated through communication of partner quality and responses to quality variation, we can state a few predictions. For an interacting pair where one has low and the other has high partner quality, the exchange of help ought to stop or be reduced to a low level after an initial higher investment by the individual with high partner quality. If both have low partner quality, little will be transacted, whereas if both have high partner quality, a more intense and lasting exchange is to be expected. Changes in partner quality during an ongoing interaction should influence the investments in similar ways.

Variation in ‘boldness’, either as part of a behavioural syndrome or as a result of variation in hunger, could be an example of partner quality variation from the carefully studied phenomenon of joint predator inspection in fish (Milinski 1987; Dugatkin & Alfieri 1991; Külling & Milinski 1992). In the interspecific mutualism between lycaenid butterfly larvae and ants, where the ants give protection and the larvae provide a nutritious secretion, both a larva's perception of the risk of enemy attack and the nutritional status of an ant colony have a strong influence on the interaction (Leimar & Axén 1993; Axén et al. 1996). For instance, a larva will sharply increase its rate of secretion following a simulated enemy attack, and the change in larval behaviour then leads to higher ant attendance. In general, empirical observations suggest that cooperative behaviour is commonly influenced by variation in needs and abilities (e.g. Heath & Tiffin 2007; Heil et al. 2009), so that variation in partner quality could be an important driving force behind cooperation.

Variation in traits such as partner quality promotes the need to respond to a partner. There is then bargaining between partners so that outcomes are negotiated. At one extreme, the negotiated outcome may be efficient in the sense that any change in outcome that increases the payoff to one partner necessarily reduces it to the other. Such an outcome is referred to as Pareto optimal. The so-called Nash bargaining solution (Muthoo 1999) is a particular example of a Pareto optimal outcome that has been highlighted by game theorists working in bargaining theory. However, whether this evolves depends on details (Binmore in press). For example, McNamara et al. (1999) presented a model of bargaining between parents over the effort expended in care of their common young for which the negotiated outcome is less cooperative than if parents had made fixed (genetically determined) care decisions.

5. Markets and choosiness

If cooperation is to evolve, individuals that are uncooperative or cheat must be at a disadvantage. Models of cooperation rely on a variety of sanctions against uncooperative individuals by other population members. One sanction is to break off the interaction with an uncooperative partner (Enquist & Leimar 1993) or to reduce investment in such a partner (Kiers et al. 2003; Kiers & Denison 2008), and punishment through aggression sometimes functions as a sanction (Bshary & Grutter 2002, 2005). When partner change can occur, there is effectively a market in partners, and a cheater is severely limited in its possibility to cheat if it cannot afford the risk of losing the partner (Bull & Rice 1991; Noë & Hammerstein 1994, 1995). However, the advantage gained by leaving an uncooperative partner depends on the availability of alternative partners and the costs of searching for a new partner (Noë & Hammerstein 1994, 1995; Bshary & Grutter 2002; Bshary & Schäffer 2002; Hammerstein & Hagen 2005; Foster & Kokko 2006). Of course, partner change is never going to be worthwhile unless better, more cooperative partners are available. Here the variation in the cooperation trait is crucial. If there is little variation, then the likelihood is that a new partner will be little different from the current partner, and the costs of partner change put individuals that are choosy about their partners at a disadvantage. In contrast, if the variation is high, then there are opportunities to find a significantly better partner if the current partner is less cooperative than the population mean. As McNamara et al. (2008) show, high variation can thus lead to the evolution of cooperation, even when the new partner is chosen at random.

In the model of McNamara et al. (2008) an individual is characterized by two genetically determined traits; how cooperative the individual is in an interaction, and how choosy it is about its partner. Each year, before breeding starts population members are paired together. Some of these pairs consist of individuals that have remained together from the previous year; those individuals without a partner from the previous year pair at random. Within each pair the individuals play each other in a game such as the Prisoner's Dilemma. The payoff in this game is a resource that individuals then use to enhance their reproductive output that year. After the game, each individual knows the cooperativeness of their partner, and must choose whether to ‘divorce’ their partner or attempt to keep the same partner until the next year. A pair only stays together if both pair members do not divorce and both survive until next year. In this model, genetic variation is maintained by mutation. McNamara et al. (2008) explore the effects of the mutation rate and the lifespan of individuals on the traits that evolve. When mutation rate is low there is little variation in the evolved population, there is little incentive to divorce, and little incentive to be cooperative. Thus, the population has low levels of both choosiness and cooperativeness. In contrast, when the mutation rate is higher there is sufficient variation to make it worth being choosy about the partner. Choosiness penalizes any lack of cooperativeness, since divorced individuals must pay the cost of finding a new partner. Thus, variation promotes cooperation. Other things being equal, increased lifespan selects for more choosiness because there is more time to find a partner that is worth retaining, and more time to exploit such a profitable partnership. Thus, increased lifespan also promotes cooperation (figure 2).

Figure 2.

Mean population effort at equilibrium in a version of the continuous Prisoner's Dilemma. Individuals play one round of the Prisoner's Dilemma each year. Efforts are shown as a function of the annual probability of mortality for different mutation rates. Mutation rate: solid line, 0.003; small-dashed line, 0.010; dotted line, 0.032; dashed-dotted line, 0.056; long-dashed line, 0.100. (Reproduced with permission from McNamara et al. 2008.)

6. Reputation and social sensitivity

It may be advantageous to be able to predict the behaviour of others when interacting with them. For example, it is useful to know whether stealing food from an individual will provoke a violent response. In humans, for instance, it may be useful to know if another is trustworthy. There is evidence that within a population individuals differ in aspects of their behaviour, but each individual is relatively consistent in its behaviour over time (Dall et al. 2004; Sih et al. 2004; Bergmüller et al. 2010). When such personality variation exists an individual that is about to interact with a social partner can use the partner's previous behaviour in order to predict the partner's behaviour in the current interaction. When this is so, it is potentially advantageous for individuals to monitor the interactions of other members of the social group; that is, to be socially sensitive. However, observing the interactions of others is liable to be costly in terms of time. Furthermore, processing the information obtained from observations may be costly in terms of the development of the required mental capacity (Brosnan et al. 2010). Thus it is reasonable to expect social sensitivity to only evolve when there is something of value to learn. Whether there is useful information to be gained by observation depends crucially on the variance in the relevant traits within the population. If this variance is low and the mean trait is stable over evolutionary time, as assumed in many evolutionary models (Lehmann & Rousset 2010), there is little useful to be learnt by observation. Instead, it is better to avoid the cost of social sensitivity and to have a genetically determined fixed behavioural strategy that is the best given that all individuals have the mean trait. In contrast, when the trait variance is high, fixed behaviour is likely to be inferior to the strategy of observing others, even though this incurs costs. These points are illustrated in McNamara et al. (2009). In their model individuals vary in their trustworthiness and social sensitivity, and these two traits coevolve. Variance in trustworthiness can be maintained by mutation. However, McNamara et al. (2009) also identify another mechanism that maintains variation. Their results show that is possible for evolution to result in a population with a bimodal distribution in the trustworthiness trait and a mixture of socially sensitive individuals and individuals that are not socially sensitive but have fixed responses. The bimodal distribution in trustworthiness provides the necessary variation to maintain a significant number of socially sensitive individuals, with others doing equally well by adopting fixed behaviours but not paying the associated observation cost. Given this mixture some individuals do well by being completely untrustworthy and exploiting individuals that have not paid the cost to observe them in previous interaction. Others do well by impressing socially sensitive individuals with their trustworthiness. The result is a bimodal distribution in trustworthiness. Thus, the variation in one trait maintains the variation in the other, and vice versa.

Johnstone (2001) also shows that individuals may vary with respect to social responsiveness. In the model presented by Johnstone each population member is involved in a series of competitive pairwise interactions. In each interaction the opponent is chosen at random from the population and each contestant chooses either to play Hawk or Dove. Three strategies are considered: always play Hawk, always play Dove and Eavesdrop. Eavesdroppers observe the last interaction of their opponent and play the best response against the action chosen by the opponent in this interaction. In this model, there are no costs of being socially sensitive. However, the population still evolves to a mixture of all three types. When Eavesdroppers are rare, the action of an opponent is well predicted by their previous action, so that it is beneficial to eavesdrop. The action taken by an Eavesdropper is not well predicted by their previous action. Thus, when Eavesdroppers become common the benefits of eavesdropping disappear. The net result is a mixture of Eavesdroppers and the other types at evolutionary stability.

7. Fighting behaviour

Animal contests are often settled by the loser withdrawing uninjured, as the result of an assessment of some aspect of fighting ability (Parker 1974). The procedures through which the contestants obtain information about their relative strengths or sizes can be seen as an example of cooperative behaviour. Through efficient assessment the payoff of the contest can come close to the maximum that would obtain if the contested resource was divided equally between the contestants without any cost (Enquist & Leimar 1983). Thus, by using assessment to predict the outcome of an all-out fight, the contestants can reduce the cost of the contest, and they might even approach a Pareto optimal solution. Even when individuals cannot assess their partner but can only assess their own ability there is reduced conflict (McNamara & Houston 2005). That assessment in contests is a cooperative endeavour can be most clearly seen when the animals use a range of different displays, some of which require the coordination of positions, movements and postures (Enquist et al. 1991). It may even be the case that some components of aggressive behaviour, such as the colour displays of cichlid fish, serve as a coordination device to facilitate the transmission of information about relative size and strength (Hurd 1997). Thus, variation in size and strength among contestants selects for various display behaviours and responses to displays that reduce the cost of fighting. In this way, fighting behaviour illustrates the general principle that variation in a trait at the population level selects for individuals to develop other traits serving to transmit, obtain and respond to this variation, in the end giving rise to mutual benefit.

Aggressive interactions that establish dominance hierarchies through winner and loser effects (Dugatkin & Druen 2004; Hsu et al. 2006) could be another example where information gained early on is used to settle future conflicts, in effect avoiding high costs of fighting. Although winner and loser effects could simply be the result of a role asymmetry established during the first encounters, it seems likely that individuals often obtain information about aspects of fighting ability or other relevant variables during the formation of a dominance hierarchy. The formation of territories through a negotiation process in which the fighting abilities influence the outcome (Pereira et al. 2003), is another example where variation and the response to variation could result in a division of a resource without excessive costs of aggression.

8. Discussion

Although some models of cooperation do not invoke the presence of variation, we have shown that it is crucial to many mechanisms that lead to the evolution of cooperative behaviour. In particular, in many social situations, the variance in a behavioural trait is important in determining how the mean value of the trait will evolve. This can occur for a range of reasons, including (i) high variation means that it is worth taking a chance that a partner is more cooperative than the mean; (ii) variation promotes individuals to learn about one another, leading to negotiation through mutual reciprocity or other forms of contingent behaviour that can produce cooperative behaviour; (iii) high variation means that it may be worth breaking off the interaction with an uncooperative partners in order to find a better partner, so putting pressure on the current partner to be cooperative; (iv) high variation means that it may be worth paying observation costs to find out about a partner, so that individuals are observed, thus putting pressure on them to be cooperative in order to establish a good reputation (see Connor 2010; Leimar & Hammerstein 2010 for detailed analyses of game-theoretic concepts).

The variation in a behavioural trait is likely to be due to both genetic variation and variation that is environmentally induced. However, in all the cases (i)–(iv) above it is the phenotypic variation that is the immediate driving force acting on behaviour. In case (ii), this variation need not have a genetic component—there will be selection to interact and negotiate whatever the cause of the variation in partner. In other circumstances it might be necessary to have a genetic component to the variation if cooperation is to evolve. So for example in case (iii), individuals that are uncooperative are penalized by having their interactions terminated, but this will only lead to a change in the level of cooperation in the population if the phenotypic variation in cooperativeness is correlated with underlying genetic variation. It seems plausible that provided there is a correlation and some genetic variation, increased developmental variation will amplify the selection pressure on cooperation, although this topic needs further analysis. Of course selection will eventually tend to reduce genetic variation. This force will be opposed by processes such as mutation and recombination. As long as sufficient phenotypic variation is maintained at the balance of these forces, and this variation has some sufficient genetic component, cooperation can emerge and be maintained.

Based on the different approaches and examples we have dealt with, something like a general recipe for successful cooperation emerges: explore and respond to variability in traits and behaviours that have an important impact on the interaction. In cases where exploration or responses are costly, there must be sufficient variability for these investments to payoff. This implies that a certain amount of cheating or exploitation can have the effect of promoting the evolution of traits that are adaptations to the variation and that may be essential for the evolutionary maintenance of cooperation. As emphasized by Enquist & Leimar (1993), evolutionary stability of cooperation may require exploitation to occur to some extent; costly defences against exploitation cannot be selected for and be maintained if exploitation never occurs. This general point has also been made by Ferrière et al. (2002) and by Foster & Kokko (2006). Just as there are many possible reasons for variability in populations in general, there are many possible reasons for the occurrence of some degree of exploitation, and empirical data show this is in fact a rather typical situation (Bronstein 2001). Modelling also indicates that a stable coexistence of mutualism and exploitation is quite feasible (Ferrière et al. 2002, 2007). Thus, it seems that the recipe could have quite broad application.

Finally, we wish to emphasize that there is great potential to empirically study the influence of different kinds of variation on cooperative behaviour, in a somewhat similar way as the influence of variation in fighting ability and resource value on fighting behaviour has been studied. There is already work of this kind (e.g. Axén et al. 1996; Heil et al. 2009), but we believe more could be done and that this could form a fruitful link between theory and observation of cooperation.