Transforming the dilemma

Abstract

How does natural selection lead to cooperation between competing individuals? The Prisoner's Dilemma captures the essence of this problem. Two players can either cooperate or defect. The payoff for mutual cooperation, R, is greater than the payoff for mutual defection, P. But a defector versus a cooperator receives the highest payoff, T, while the cooperator obtains the lowest payoff, S. Hence, the Prisoner's Dilemma is defined by the payoff ranking T > R > P > S. In a well-mixed population, defectors always have a higher expected payoff than cooperators, and therefore natural selection favors defectors. The evolution of cooperation requires specific mechanisms. Here we discuss five mechanisms for the evolution of cooperation: direct reciprocity, indirect reciprocity, kin selection, group selection and network reciprocity (or graph selection). Each mechanism leads to a transformation of the Prisoner's Dilemma payoff matrix. From the transformed matrices, we derive the fundamental conditions for the evolution of cooperation. The transformed matrices can be used in standard frameworks of evolutionary dynamics such as the replicator equation or stochastic processes of game dynamics in finite populations.

1. Introduction

Evolutionary biologists are fascinated by cooperation. We think this fascination is entirely justified, because cooperation is essential for construction. Whenever evolution ‘constructs’ a new level of organization, cooperation is involved. The very origin of life, the emergence of the first cell, the rise of multi-cellular organisms, and the advent of human language are all based on cooperation. A higher level of organization emerges, whenever the competing units on the lower level begin to cooperate. Therefore, we propose that cooperation is a third fundamental principle of evolutionary dynamics besides mutation and selection. Evolution occurs in populations of reproducing individuals. Inaccurate reproduction can lead to mutation. Mutation can lead to selection. Selection can lead to cooperation.

Cooperation is always vulnerable to exploitation by defectors. Hence, the evolution of cooperation requires specific mechanisms, which allow natural selection to favor cooperation over defection. In this paper, we discuss five such mechanisms, and for each mechanism we derive the fundamental condition for the evolution of cooperation.

The meaning of the word ‘cooperation’ in evolutionary biology is more specific than just ‘working together’. In the narrow sense, ‘cooperation’ and ‘defection’ are the two possible actions that are defined by the Prisoner's Dilemma. The payoff matrix of the Prisoner's Dilemma (Rapoport and Chammah 1965; Axelrod 1984) is given by

$$\begin{array}{l}\begin{array}{c}CD\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}R& S\\ T& P\end{array}\right)\end{array}$$ (1.1)

The entries of the payoff matrix refer to the row player. If a cooperator, C, interacts with another cooperator, both get payoff R, which is the ‘reward for mutual cooperation. If a cooperator, C, meets a defector, D, the cooperators gets the ‘sucker's payoff’, S, while the defector gets the highest payoff of the game, T, which denotes the ‘temptation of defect’. Two defectors obtain the payoff P, which stands for the ‘punishment’ of mutual defection. The game is a Prisoner's Dilemma if

$$T>R>P>S.$$ (1.2)

The payoff matrix (1.1) together with the ranking (1.2) can be seen as mathematical definitions of ‘cooperation’ and ‘defection’.

In a broader sense, ‘cooperation’ can be used as referring to an action where one individuals pays a cost for another individual to receive a benefit. But this definition alone need not lead to a Prisoner's Dilemma. It is, for example, also compatible with a snow-drift game (Hauert and Doebeli 2004; Hauert et al. 2006; Nowak 2006a), where cooperators and defectors can coexist without the need for a mechanism for the evolution of cooperation. In this article, we use ‘cooperation’ and ‘defection’ in the narrow sense as defined by the context of an underlying Prisoner's Dilemma game.

In the Prisoner's Dilemma, defectors dominate cooperators unless a mechanism for the evolution of cooperation is at work. We will discuss five mechanisms for the evolution of cooperation: direct reciprocity, indirect reciprocity, kin selection, group selection and network reciprocity (or graph selection). Direct reciprocity is the idea that there are repeated encounters between the same two individuals: my action depends on what you have done to me in previous encounters. For indirect reciprocity there are repeated encounters in a population of individuals: my action depends on what you have done to me and to others. Kin selection occurs whenever games are played between genetically relatives. Group selection refers to the mechanism where competition not only occurs between individuals, but also between groups. Network reciprocity operates in structured populations, where cooperators can prevail over defectors by forming clusters.

All five approaches have led to different mathematical investigations, but here we show that the essential properties of each mechanism can be described by a transformation of the Prisoner's Dilemma payoff matrix. From these transformed matrices we can immediately derive the fundamental conditions for the evolution of cooperation. Our analysis reveals how each mechanism operates at a fundamental level. But we do not claim that every aspect of each mechanism can be studied in the same way. For example, very different calculations are needed to find good strategies for direct or indirect reciprocity (Nowak and Sigmund 1992, 1993; Ohtsuki and Iwasa 2004, 2006). Furthermore, the population structure of group selection is entirely different to that of network reciprocity. In our opinion it is also incorrect to argue that kin selection refers to the same process as group selection. Inclusive fitness theory leads to fairly general methods for analyzing evolutionary dynamics, but does not imply that the driving force for all evolution of cooperation is kin selection. We will revisit this point in the discussion.

This paper is an extension of earlier work (Nowak 2006b), which investigates a simplified Prisoner's Dilemma given by two parameters, b and c, denoting respectively the benefit and cost of cooperation. In this case, we have R = b − c, S = −c, T = b and P = 0, which implies the restriction R + P = T + S. In contrast, the general Prisoner's Dilemma, which is studied here, has four parameters, displays a richer dynamical behavior and leads to generalizations of the fundamental rules presented in (Nowak 2006b). Moreover, for kin selection, group selection and network reciprocity, the full Prisoner's Dilemma allows coexistence between cooperators and defectors if R + P < T + S and bi-stability if R + P > T + S. Neither of these outcomes are possible for the simplified Prisoner's Dilemma with R + P = T + S.

We begin with some simple remarks on evolutionary game dynamics. Then we discuss each of the five mechanisms. Finally, we summarize the key findings and discuss their limitations.

2. Evolutionary Game Dynamics

Evolutionary game dynamics describe frequency dependent selection. The fitness of an individual is not constant, but depends on the relative abundance (= frequency) of various strategies (= phenotypes) in the population. The outcome of the game is related to reproductive success. Payoff determines fitness. Reproduction can be genetic or cultural. Genetic reproduction means that individuals leave genetic offspring which inherit their strategy. Cultural reproduction means that individuals are imitated by others; thereby strategies reproduce by imitation or learning. The first ideas of evolutionary game theory appeared in papers by Hamilton (1964), Trivers (1971) and Maynard Smith and Price (1973). Books on evolutionary game theory include (Maynard Smith 1982; Weibull 1995; Samuelson 1997; Hofbauer and Sigmund 1998; Gintis 2000; Cressman 2003; Nowak 2006a). For recent reviews see (Hofbauer and Sigmund 2003; Nowak and Sigmund 2004). Evolutionary game theory is a general approach to evolutionary dynamics with constant selection being a special case.

Consider a game between two strategies, A and B, given by the payoff matrix

$$\begin{array}{l}\begin{array}{c}AB\end{array}\\ \begin{array}{c}A\\ B\end{array}\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)\end{array}$$ (2.1)

The entries denote the payoffs for the row player. Thus, strategy A obtains payoff a when playing another A player, but payoff b when playing a B player. Likewise, strategy B obtains payoff c when playing an A player and payoff d when playing a B player.

If a > c and b > d, then A dominates B. In this case, it is always better to use strategy A. The expected payoff of A players is greater than that of B players for any composition of a well-mixed population. A is an unbeatable strategy in the sense of (Hamilton 1967). If instead a < c and b < d, then B dominates A and we have exactly the reverse situation (Fig 1).

Figure 1. Evolutionary dynamics of a two-player, two-strategy game.

The entries of the matrix specify the payoff for the row player. There are four possibilities: (i) if a > c and b > d, then strategy A dominates strategy B; (ii) if a > c and b < d, then both A and B are evolutionarily stable strategies (ESS); the game is bi-stable and is called a ‘coordination game’; (iii) if a < c and b > d, then both strategies are best replies to each other, and we have a Hawk-Dove game, where both strategies coexist; (iv) if a < c and b < d, then B dominates A.

If a > c and b < d, then both strategies are best replies to themselves, which leads to a ‘coordination game’. In a population where most players use A, it is best to use A. In a population where most players use B, it is best to use B. A coordination game leads to bi-stability: both strategies are stable against invasion by the other strategy.

If a < c and b > d, then both strategies are best replies to each other, which leads to a ‘Hawk-Dove game’ (Maynard Smith 1982). In a population where most players use A, it is best to use B. In a population where most players use B, it is best to use A. A Hawk-Dove game leads to co-existence between the two strategies.

If a > c then A is a strict Nash equilibrium. Likewise, if b < d then B is a strict Nash equilibrium. A strategy which is a strict Nash equilibrium is always an evolutionarily stable strategy (ESS). An ESS is stable against invasion by a small fraction of mutants using the other strategy in an infinitely large, well-mixed population (Maynard Smith 1982). In the terminology of evolutionary graph theory (Lieberman et al. 2005), a well-mixed population is given by a complete graph with identical weights: any two individuals are equally likely to interact with each other. Note that the ESS condition does not imply protection by selection in a finite population (Nowak et al. 2004).

3. Direct Reciprocity

In a non-repeated Prisoner's Dilemma, it is best to defect no matter which strategy is adopted by the other player (because R < T and S < P). But if there are repeated encounters between the same two individuals, then direct reciprocity can emerge and lead to the evolution of cooperation (Trivers 1971, 1985). Direct reciprocity is based on the idea ‘I help you and you help me.’ In each round the two players must choose between cooperation and defection. With probability w there is another round. With probability 1 − w the game is over. Hence, the average number of interactions between two individuals is 1/(1 − w).

There are many conceivable deterministic and stochastic strategies for the repeated Prisoner's Dilemma (Axelrod and Hamilton 1981; Axelrod 1984; Milinski 1984; Molander 1985; May 1987; Kraines and Kraines 1989; Fudenberg and Maskin 1990; Nowak and Sigmund 1990, 1992, 1993; Imhof et al. 2005), and the game can be played with simultaneous or alternating moves (Nowak and Sigmund 1994). But for the purpose of this paper we only need to consider two very simple strategies. Our defectors, D, defect in every move. Our cooperators, C, play tit-for-tat: they start with a cooperation and then do whatever the other player has done in the previous move.

If two cooperators meet, they cooperate all the time. If two defectors meet, they defect all the time. If a cooperator meets a defector, the cooperator cooperates in the first round and defects afterwards, while the defector defects in every round. The payoff matrix is given by

$$\begin{array}{l}\begin{array}{c}CD\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}\frac{R}{1-w}& S+\frac{wP}{1-w}\\ T+\frac{wP}{1-w}& \frac{P}{1-w}\end{array}\right)\end{array}$$ (3.1)

Note that defectors, D, are always ESS (because P > S). Cooperators are ESS if

$$w>\frac{T-R}{T-P}.$$ (3.2)

Hence there are two possibilities: (i) if inequality (3.2) holds then both cooperation and defection are ESS, and the game is a coordination game; (ii) if inequality (3.2) does not hold then defection dominates cooperation (Fig 2). Therefore, inequality (3.2) represents a minimum requirement for the evolution of cooperation. If there are sufficiently many rounds, then direct reciprocity can lead to the evolution of cooperation. This argument is related to the Folk theorem (Fudenberg and Maskin 1986; Binmore 1991; Fudenberg and Tirole 1991).

Figure 2. Direct and indirect reciprocity can lead to the evolution of cooperation.

While direct reciprocity is based on repeated encounters between the same two individuals, indirect reciprocity uses the experience of others. Defectors are always ESS. Cooperators are ESS if w or q exceed (T − R)/(T − P), where w is the probability of another round in direct reciprocity, and q is the coefficient of social acquaintanceship under indirect reciprocity.

4. Indirect Reciprocity

As Richard Alexander (1987) has noted, ‘indirect reciprocity arises out of direct reciprocity in the presence of interested audiences’. For direct reciprocity, my decision is based on what you have done to me in previous encounters. For indirect reciprocity my decision is also based on what you have done to others. Indirect reciprocity represents the concept ‘I help you and somebody will help me.’ Indirect reciprocity is based on reputation (Nowak and Sigmund 1998a). Each event can be seen as an interaction between two people, according to a single Prisoner's Dilemma game given by payoff matrix (1.1). Each game is observed by others. Cooperation is costly, but leads to the reputation of being a helpful individual. Defection is more profitable in the short run, but leads to a bad reputation. Natural selection favors strategies that base their decision to cooperate or to defect on the reputation of oneself and of others (Nowak and Sigmund 1998a,b, 2005; Lotem et al. 1999; Fishman et al. 2001; Leimar and Hammerstein 2001; Fishman 2003; Panchanathan and Boyd 2003, 2004; Takahashi and Mashima 2003; Brandt and Sigmund 2004, 2005; Ohtsuki and Iwasa 2004, 2006, 2007). Experimental studies confirm that helpful individuals are more likely to receive help in the future (Wedekind and Milinski 2000; Dufwenberg et al. 2001; Milinski et al. 2002, 2006; Engelmann and Fischbacher 2002; Wedekind and Braithwaite 2002; Bolton et al. 2005; Rockenbach and Milinski 2006).

In order to derive a necessary condition for the evolution of cooperation by indirect reciprocity, let us study the interaction between two basic strategies: (i) defectors who always defect and (ii) cooperators who cooperate unless they know the reputation of the other person to indicate a defector. The parameter q denotes the probability to know the reputation of another individual. A cooperator always cooperates with another cooperator, but cooperates with a defector only with probability 1−q. Defectors never cooperate. We obtain the payoff matrix

$$\begin{array}{l}\begin{array}{c}CD\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}R& \left(1-q\right)S+qP\\ \left(1-q\right)T+qP& P\end{array}\right)\end{array}$$ (4.1)

This payoff matrix leads to exactly the same analysis as we have performed for direct reciprocity in the previous section. We obtain the same results, with q instead of w (Fig 2). The probability to know the reputation of another player must exceed a certain threshold value,

$$q>\frac{T-R}{T-P}.$$ (4.2)

If this condition holds, then indirect reciprocity can lead to the evolution of cooperation.

5. Kin Selection

The concept of kin selection arose from the idea that evolutionary games are often played between individuals who are genetic relatives (Hamilton 1964, 1996; Maynard Smith 1964; Cavalli-Sforza and Feldman 1978; Grafen 1979, 1985; Queller 1985, 1992; Frank 1998; Taylor 1996; Taylor and Frank 1996; Rousset and Billiard 2000; West et al. 2002; Griffin and West 2003; Foster et al. 2006; Taylor et al. 2007). A gene which encodes altruistic behavior toward another individual promotes its own survival, if it is also present in the recipient of the altruistic act. The ‘state of the art’ approach is to calculate relatedness using population dynamics (Griffin and West 2002; Taylor et al. 2007; Wild and Taylor 2004; Wild and Traulsen 2007) and the Price equation (Price 1970; Taylor 1989). Another approach is called ‘direct fitness’ (Taylor and Frank 1996; Taylor et al. 2007).

But for the purpose of this paper, we use a method that was originally proposed by Maynard Smith (1982) for the Hawk-Dove game. Consider a population where the average relatedness between interacting individuals is given by r, which is a number between 0 and 1. The payoff received by the other player is added to your own payoff multiplied by r. The sum is divided by 1 + r in order to keep the total payoff between two players constant. Therefore, we obtain the modified payoff matrix

$$\begin{array}{l}\begin{array}{c}CD\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}R& \frac{S+rT}{1+r}\\ \frac{T+rS}{1+r}& P\end{array}\right)\end{array}$$ (5.1)

Cooperators are ESS if

$$r>r_C=\frac{T-R}{R-S}.$$ (5.2)

Defectors are ESS if

$$r<r_D=\frac{P-S}{T-P}.$$ (5.3)

The evolutionary outcome depends on the relative ranking of r, r_C and r_D. We must distinguish two parameter regions (Fig 3).

Figure 3. Kin selection, group selection and network reciprocity can lead to the evolution of cooperation.

For kin selection, the parameter r is the coefficient of genetic relatedness between individuals. For group selection, the parameters m and n denote respectively the number of groups and the number of individuals per group (group size). For network reciprocity, we have F_C = (T − R)k² − (T − P)k + (R + P − T − S) and F_D = (P − S)k² − (R − S)k − (R + P − T − S), where k is the degree of the graph (that is the number of neighbors of each individual). For all three models we find: if R + P > T + S, then cooperators and defectors cannot co-exist; if R + P < T + S, then cooperators and defectors cannot simultaneously be ESS.

I. If R + P > T + S then r_D > r_C. There are three possibilities: (i) if r_D > r_C > r then defectors dominate; (ii) if r_D > r > r_C then both cooperators and defectors are ESS; (iii) if r > r_D > r_C then cooperators dominate.

II. If R + P < T + S then r_C > r_D. Again there are three possibilities: (i) if r_C > r_D > r then defectors dominate; (ii) if r_C > r > r_D then neither cooperators nor defectors are ESS; (iii) if r > r_C > r_D then cooperators dominate.

In the degenerate case, R + P = T + S, we have r_C = r_D. Then either C dominates (when r > r_C) or D dominates (when r < r_D). Neither coexistence, nor bi-stability between C and D are possible. The case R + P = T + S is called ‘equal gains from switching’ (Nowak and Sigmund 1990): when switching from C to D while playing against C the gain is T − R and while playing against D the gain is P − S. The condition T − R = P − S implies that the two gains are equal. The two-parameter Prisoner's Dilemma, where cooperators pay a cost, c, for the other person to receive a benefit, b, leads to equal gains from switching, because T + b, R = b − c, P = 0 and S = −c (Nowak 2006b). In this special case, inequalities (4.2) and (4.3) lead to r > c/b, which is Hamilton's rule.

6. Group Selection

Group selection is based on the idea that competition occurs not only between individuals but also between groups. Many models of group selection have been proposed over the years (Wynne-Edwards 1962; Williams 1966; Wilson 1975; Maynard Smith 1976; Slatkin and Wade 1978; Uyenoyama and Feldman 1980; Crow and Aoki 1982; Leigh 1983; Wilson 1983; Harvey et al. 1985; Nunney 1985; Harpending and Rogers 1987; Szathmáry and Demeter 1987; Michod 1999; Boyd and Richerson 2002; Kerr and Godfrey-Smith 2002; Paulsson 2002; Fletcher and Zwick 2004; Traulsen et al. 2005; Wilson and Hölldobler 2005; Bowles 2006; Killingback et al. 2006). Here we use an approach described by (Traulsen and Nowak 2006). A population is subdivided into m groups. The maximum size of a group is n. Individuals interact with others in the same group according to a Prisoner's Dilemma. The fitness of an individual is 1 − ω + ωF, where F is the payoff and ω the intensity of selection. At each time step, an individual from the entire population is chosen for reproduction proportional to fitness. The offspring is added to the same group. If the group reaches the maximum size, it can split into two groups with a certain probability, p. In this case, a randomly selected group dies to prevent the population from exploding. The maximum population size is mn. With probability 1 − p the group does not divide. In this case, a random individual of that group is chosen to die. For small p, the fixation probability of a single cooperator in the entire population is given by the fixation probability of a single cooperator in a group times the fixation probability of that group. The model can be extended to include migration (Traulsen and Nowak 2006), but this is not needed here.

The payoff matrix that describes the interactions between individuals of the same group is given by

$$\begin{array}{l}\begin{array}{c}CD\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}R& S\\ T& P\end{array}\right)\end{array}$$ (6.1)

Between groups there is no game dynamical interaction in our model, but groups divide at rates that are proportional to the average fitness of individuals in that group. The multi-level selection is an emerging property of the population structure. Therefore one can say that cooperator groups have a constant payoff R, while defector groups have a constant payoff P. Hence, in a sense the following ‘game’ of constant selection describes the competition between groups

$$\begin{array}{l}\begin{array}{c}CD\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}R& R\\ P& P\end{array}\right)\end{array}$$ (6.2)

Again the ‘fitness’ of a group is 1 − ω + ωF where F is its ‘payoff’. Surprisingly, for weak selection (w ≪ 1) and large n and m, the essence of the overall selection dynamics on two levels can be described by a single payoff matrix, which is the sum of matrix (6.1) multiplied by the group size, n, and matrix (6.2) multiplied by the number of groups, m (Nowak 2006b). The result is

$$\begin{array}{l}\begin{array}{c}CD\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}\left(n+m\right)R& nS+mR\\ nT+mP& \left(n+m\right)P\end{array}\right)\end{array}$$ (6.3)

The intuition for adding the two matrices multiplied with the respective population size is as follows. For fixation of a new strategy in a homogeneous population using the other strategy, first the game dynamics within one group of size n have to be won and then the game dynamics between m groups have to be won. For weak selection and large m and n, the overall fixation probability is the same as the fixation probability in the single game using the combined matrix 6.3 and population size, mn. We derive the addition formula of matrix 6.3 in Appendix A. Therefore, some aspects of the stochastic selection process on two levels can be studied by a standard replicator equation using the combined matrix.

From matrix (6.3) we see that cooperators are ESS if

$$\frac{m}{m+n}>\frac{T-R}{T-P}.$$ (6.4)

Defectors are ESS if

$$\frac{m}{m+n}<\frac{P-S}{R-S}.$$ (6.5)

Again there are two parameter regions defined by R + P > T + S and R + P < T + S. The same six cases apply as for kin selection, but the thresholds have different values (Fig 3).

7. Network Reciprocity

Spatial games can lead to cooperation in the absence of any strategic complexity : cooperators can coexist with defectors and sometimes even outcompete them (Nowak and May 1992). This effect is called ‘spatial reciprocity’. Spatial games are usually played on regular lattices such as square, triangular or hexagonal lattices (Nowak and May 1992; Herz 1994; Rand and Wilson 1995). Evolutionary graph theory (Lieberman et al. 2005) is a general approach to study the effect of population structure or social networks on evolutionary or ecological dynamics (Durrett and Levin 1994; Hassell et al. 1994; Nakamaru et al. 1997, 1998; Skyrms and Pemantle 2000; Abramson and Kuperman 2001; Ebel and Bornholdt 2002; Hauert and Doebeli 2004; Rousset 2004; Santos and Pacheco 2005; Szabó et al. 2005; Vukov and Szabó 2005; May 2006; Santos et al. 2006; Vukov et al. 2006; Wu et al. 2006). The individuals of a population occupy the vertices of a graph. The edges denote who interacts with whom. In principle, there can be two different graphs (Ohtsuki et al. 2007): the ‘interaction graph’ determines who plays with whom; the ‘replacement graph’ determines who competes with whom for reproduction. Here we assume that the interaction and replacement graphs are identical. Network reciprocity is a generalization of spatial reciprocity to graphs (Ohtsuki et al. 2006): on graphs cooperators form clusters which can enable them to outcompete defectors.

We consider a ‘two coloring’ of the graph: each vertex is either a cooperator or a defector. Each individual interacts with all of its neighbors according to the standard payoff matrix (1.1). The payoffs are added up. The fitness of an individual is given by 1 − ω + ωF where ω ∈ [0, 1] denotes the intensity of selection and F denotes the payoff for this individual. Here we consider evolutionary dynamics according to death-birth updating (Ohtsuki et al. 2006): in each time step a random individual is chosen to die; then the neighbors compete for the empty site proportional to their fitness. There are many different updating mechanisms for games on graphs and some of them lead to very different results (Nakamaru et al. 1997, 1998; Ohtsuki et al. 2006). As yet there is no classification of all possible updating rules on graphs.

A calculation using pair approximation on regular graphs (where each vertex has k edges) leads to a deterministic differential equation which describes how the expected frequency of cooperators (and defectors) changes over time (Ohtsuki and Nowak 2006a). This differential equation turns out to be a standard replicator equation (Taylor and Jonker 1978; Hofbauer et al. 1979; Zeeman 1980) with a modified payoff matrix. For the interaction between cooperators and defectors on a graph with degree k > 2, the modified payoff matrix is of the form

$$\begin{array}{l}\begin{array}{c}CD\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}R& S+H\\ T-H& P\end{array}\right)\end{array}$$ (7.1)

where

$$H=\frac{\left(k+1\right)\left(R-P\right)-T+S}{\left(k+1\right)\left(k-2\right)}.$$ (7.2)

For a derivation of this transformation see (Ohtsuki and Nowak 2006a). Cooperators are ESS if R > T − H or

$$F_C=k^2\left(T-R\right)-k\left(T-P\right)+\left(R+P-T-S\right)<0.$$ (7.3)

Defectors are ESS if S + H < P or

$$F_D=k^2\left(P-S\right)-k\left(R-S\right)+\left(T+S-R-P\right)>0.$$ (7.4)

Note that

$$F_C-F_D=(T+S-P-R)(k^2-k-2).$$

We have two parameter regions when k > 2 (Fig 3):

I. If R + P > T + S then F_D > F_C. There are three possibilities: (i) if F_D > F_C > 0 then defectors dominate; (ii) if F_D > 0 > F_C then both cooperators and defectors are ESS; (iii) if 0 > F_D > F_C then cooperators dominate.

II. If R + P < T + S then F_C > F_D. Again there are three possibilities: (i) if 0 > F_C > F_D then cooperators dominate; (ii) if F_C > 0 > F_D then neither cooperators nor defectors are ESS; (iii) if F_C > F_D > 0 then defectors dominate.

In the degenerate case, R + P = T + S, we have F_C = F_D and therefore either cooperators or defectors dominate; neither bi-stability nor coexistence are possible. The same is true for graphs with k = 2 (Ohtsuki and Nowak 2006b).

8. Conclusion

We have studied five mechanisms for the evolution of cooperation. Each mechanism leads to a transformation of the Prisoner's Dilemma payoff matrix. From the transformed matrices, we have derived the fundamental condition for each mechanism to facilitate the evolution of cooperation. The transformed matrices can also be used to study evolutionary success in finite populations (Nowak 2006a,b).

Direct reciprocity can lead to evolution of cooperation if

$$w>\frac{T-R}{T-P}.$$ (8.1)

The parameter w denotes the probability of playing another round in the repeated Prisoner's Dilemma game.

Indirect reciprocity can lead to evolution of cooperation if

$$q>\frac{T-R}{T-P}.$$ (8.2)

The parameter q denotes the probability of knowing the reputation of the other person. It is a measure for the degree of social acquaintanceship in the population.

Kin selection can lead to evolution of cooperation if

$$r>\text{min}\{\frac{T-R}{R-S},\frac{P-S}{T-P}\}.$$ (8.3)

The parameter r denotes the fraction of the payoff of the other person that is added to my payoff. It can be seen as a measure for genetic relatedness in the population. The notation ‘min{x, y}’ means to take the smaller value of either x or y. In our case, this notation is a simple way to cover the two cases R + P > T + S and R + P < T + S.

Group selection can lead to evolution of cooperation if

$$\frac{m}{m+n}>\text{min}\{\frac{T-R}{T-P},\frac{P-S}{R-S}\}.$$ (8.4)

The parameters m and n denote respectively the number of groups and the group size.

Network reciprocity can lead to evolution of cooperation if

$$\text{min}\left\{F_C,F_D\right\}<0,$$ (8.5)

where

$$F_C=k^2\left(T-R\right)-k\left(T-P\right)+\left(R+P-T-S\right)$$ (8.6)

$$F_D=k^2\left(P-S\right)-k\left(R-S\right)+\left(T+S-R-P\right).$$ (8.7)

The parameter k is the degree of the regular graph.

For kin, group and graph selection, the original Prisoner's Dilemma matrix is tranformed into a new matrix

$$\begin{array}{l}\begin{array}{c}CDCD\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}R& S\\ T& P\end{array}\right)\to \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}R\prime & S\prime \\ T\prime & P\prime \end{array}\right)\end{array}$$ (8.8)

given by 5.1, 6.3, and 7.1, respectively. We note that for each of the transformed matrices, the sign of R′ + P′ − S′ − T′ is the same as the sign of R + P − S − T. Therefore, the sign of R + P − S − T determines which evolutionary outcomes are possible for the transformed payoff matrix. If R + P − T − S > 0, then cooperators and defectors cannot simultaneously be ESS in the transformed game. If R + P − T − S < 0, then cooperators and defectors cannot coexist in the transformed game. If R + P − T − S = 0, then either cooperators or defectors must dominate.

For direct and indirect reciprocity, conditional strategies are needed. In contrast, kin selection, group selection and network reciprocity can support the evolution of unconditional cooperators.

Our approach has certain limitations which should be discussed. We have studied particular models of kin selection, group selection and network reciprocity. For each mechanism, other models should be investigated and might lead to (slightly) different results. We wish to think, however, that we have chosen the simplest possible approaches and that our findings reflect the essence of each mechanism. Adding further complexity is always possible and sometimes useful.

We might not have discussed all possible mechanisms for the evolution of cooperation. For example, it is unclear at present if ‘green beards’ constitute an independent mechanism for the evolution of cooperation (Keller and Ross 1998; Riolo et al. 2001; Traulsen and Schuster 2003; Traulsen and Nowak 2007) or require a spatial setting ( = network reciprocity) (Jansen and van Baalen 2006). Active linking on graphs (Pacheco et al. 2006) is a powerful method to get cooperation, but can be viewed as a special form of direct reciprocity: cooperators have longer lasting interactions than defectors. Volunteering is the idea that individuals can choose to participate in a Prisoner's Dilemma or remain loners (Hauert et al. 2002). Volunteering leads to a relaxed social dilemma where some frequency of cooperation is (neutrally) stable even in well-mixed populations. Volunteering can also stabilize punishment (Fowler 2005; Brandt al. 2005; Hauert et al. 2007).

Punishment (Yamagishi 1986; Clutton-Brock and Parker 1995; Fehr and Gächter 2002; Sigmund et al. 2001; Camerer and Fehr 2006; Rockenbach and Milinski 2006), sometimes also called ‘strong reciprocity’ (Boyd et al. 2003; Bowles and Gintis 2004), is not a mechanism for the evolution of cooperation, but an extension of the basic game from two possible actions (cooperation and defection) to three (cooperation, defection and punishment). All models of punishment are either based on direct or indirect reciprocity: if I punish someone who has defected with me, I use direct reciprocity; if I punish someone who has defected with others, I use indirect reciprocity.

Indirect reciprocity is the key mechanism for understanding the evolution of any pro-social behavior in humans. Humans obviously take a keen interest in following social interactions between other people and make decisions based on what has happened between others. Indirect reciprocity provides the right selection pressure for both social intelligence and human language.

It is sometimes proposed that kin selection and group selection refer to the same mechanism for the evolution of cooperation (Dugatkin and Reeve 1994; Lehmann and Keller 2006; Ratnieks 2006; West et al. 2007; Lehmann et al. 2007). Although there could be overlap between the two mechanisms in some cases, we do not consider this to be a useful perspective, in general. For example, kin selection could lead to cooperation in well-mixed populations, if there is some form of kin recognition. Such a process has nothing in common with group selection. Vice versa, there can be group selection in the setting of cultural evolutionary dynamics, where the members of each group are not genetic relatives. It would be strange to call such a process ‘kin selection’. In our opinion, the term kin selection should refer exclusively to situations where an evolutionary game occurs between genetic relatives (as originally proposed by Maynard Smith, 1964).

Inclusive fitness theory has given rise to fairly general methods for calculating evolutionary dynamics (Grafen 1985, 2006; Taylor 1989; Frank 1998; Michod 1999; Rousset and Billiard 2000; Rousset 2004; Lehmann et al. 2007; Wild and Traulsen 2007). These methods can be used to derive certain aspects of direct reciprocity (Fletcher and Zwick 2006), group selection (Lehmann et al. 2007; Wild and Traulsen 2007) and network reciprocity (Taylor et al. 2007). But inclusive fitness itself is not a mechanism for the evolution of cooperation. Instead it is a method for calculating evolutionary dynamics. For example, inclusive fitness calculations can also be used to study the evolution of spiteful behavior (Hamilton 1970; Gardner and West 2004a,b). Wilson (2007) points out that inclusive fitness is only a ‘measure’ but gives no account of the selective forces that drive eusocial evolution. Similarly, the methods of evolutionary game theory can be used to analyze all aspects of the evolution of cooperation, but nobody would claim that evolutionary game theory itself is a mechanism for the evolution of cooperation.

In a recent paper, Lehmann et al (2007) have used inclusive fitness theory to rederive some results of the group selection model proposed by Traulsen and Nowak (2006). While it is always interesting to have different derivations for the same result, the conclusion of Lehman et al (2007) that group selection and kin selection are the same process does not follow from their work. Furthermore, Lehmann et al (2007) claim incorrectly that they have derived all results of Traulsen and Nowak (2006), which is not the case, because Lehmann et al (2007) only investigate the simplified Prisoner's Dilemma, while Traulsen and Nowak (2006) derive results for a general payoff matrix. This is an important difference, because Lehmann et al (2007) base their work on the approach of (Rousset and Billiard 2000), which is only valid for the degenerate case R + P = T + S, see (Wild and Traulsen 2007).

If one wishes to search for a unifying principle that applies to all five mechanisms for the evolution of cooperation, which are discussed here, then one can point to assortment. Each mechanism must achieve that cooperators are more likely to cooperate with each other than with defectors. Assortment can be reached by repeated interactions between two individuals (direct reciprocity), monitoring social interactions (indirect reciprocity), kin recognition (kin selection), competition between groups (group selection) or cluster formation (network reciprocity, graph selection). Assortment, however, is not a mechanism for evolution of cooperation, but the consequence of a mechanism. The crucial question is always how assortment is achieved.

In summary, we have presented five mechanisms for the evolution of cooperation. For each mechanism we derive a transformation of the Prisoner's Dilemma payoff matrix. The transformed matrices lead to five fundamental rules for the evolution of cooperation.

Figure 4. Each mechanism can be described by a transformation of the Prisoner's Dilemma payoff matrix.

The transformed matrices lead directly to the crucial conditions for the evolution of cooperation. Cooperation can evolve, if either cooperators are ESS or defectors are not ESS (that is whenever defectors do not dominate cooperators). For network reciprocity we have used the abbreviation H = [(k + 1)(R − P) − T + S]/[(k + 1)(k − 2)] where k > 2 is the degree of the graph (number of neighbors per individual).

Appendix A. The addition formula for group selection

In the limit of slow group division, most groups have size n and are homogeneous when they divide. Hence, the fixation probability ρ_C of a single cooperator is the product of the fixation probability within the group φ_C, and the fixation probability of the cooperator group Φ_C (Traulsen and Nowak 2006).

Under the assumptions of weak selection (w ≪ 1), large group size n (n ≫ 1) and large number of groups m (m ≫ 1), we find that

$$\begin{array}{ll}{\phi }_C\simeq \frac{1}{n}+\frac{w}{6}(R-T+2(S-P)),& {\Phi }_C\end{array}\simeq \frac{1}{m}+\frac{w}{6}(3(R-P)).$$ (A.1)

Within a group, interacting individuals play the game given by matrix 6.1, and between groups, the game is described by matrix 6.2.

Therefore,

$$\begin{array}{l}{\rho }_C\simeq \frac{1}{mn}+\frac{w}{6}\left(\frac{1}{m}(R-T+2(S-P))+\frac{3}{n}(R-P)\right)\\ =\frac{1}{mn}+\frac{w}{6}\left(\frac{n}{mn}(R-T+2(S-P))+3\frac{m}{mn}(R-P)\right)\\ =\frac{1}{mn}+\frac{w}{6}\left(\frac{n+m}{mn}R-\frac{nT+mP}{mn}+2(\frac{nS+mR}{mn}-\frac{n+m}{mn})P)\right)\end{array}$$ (A.2)

Equation A.2 is simply the fixation probability for weak selection of a single A player in a population of size mn engaged in the game

$$\begin{array}{l}\begin{array}{c}AB\end{array}\\ \begin{array}{c}A\\ B\end{array}\left(\begin{array}{cc}\frac{\left(n+m\right)R}{mn}& \frac{nS+mR}{mn}\\ \frac{nT+mP}{mn}& \frac{\left(n+m\right)P}{mn}\end{array}\right).\end{array}$$ (A.3)

The constant factor 1/mn in the matrix A.3 only weakens the selection and has no effect on the sign of ρ_C − 1/(mn).