Direct reciprocity on graphs

Abstract

Direct reciprocity is a mechanism for the evolution of cooperation based on the idea of repeated encounters between the same two individuals. Here we examine direct reciprocity in structured populations, where individuals occupy the vertices of a graph. The edges denote who interacts with whom. The graph represents spatial structure or a social network. For birth-death or pairwise comparison updating, we find that evolutionary stability of direct reciprocity is more restrictive on a graph than in a well-mixed population, but the condition for reciprocators to be advantageous is less restrictive on a graph. For death-birth and imitation updating, in contrast, both conditions are easier to fulfill on a graph. Moreover, for all four update mechanisms, reciprocators can dominate defectors on a graph, which is never possible in a well-mixed population. We also study the effect of an error rate, which increases with the number of links per individual; interacting with more people simultaneously enhances the probability of making mistakes. We provide analytic derivations for all results.

1. Introduction

Cooperation means that one individual pays a cost, c, for another individual to receive a benefit, b. Cost and benefit are measured in terms of reproductive success. Reproduction can be genetic or cultural. In the absence of a specific mechanism for evolution of cooperation, natural selection favors defectors. There are at least five mechanisms that can lead to the evolution of cooperation: kin selection, group selection, direct reciprocity, indirect reciprocity and network reciprocity. In this paper, we study the interaction between direct and network reciprocity.

Direct reciprocity is based on the idea of repeated encounters between two individuals (Trivers 1971) according to the principle, ‘I scratch your back and you scratch mine.’ The game theoretic framework of direct reciprocity is the repeated Prisoner's Dilemma, which has been the subject of numerous studies across various disciplines (Rapoport & Chammah 1965, Axelrod & Hamilton 1981, Axelrod 1984, Selten & Hammerstein 1984, Milinski 1987, May 1987, Axelrod & Dion 1988, Fudenberg & Maskin 1990, Imhof et al 2005). A large number of strategies for playing the repeated PD have been analyzed. The most prominent ones are tit-for-tat (Axelrod 1984), generous-tit-for-tat (Nowak & Sigmund 1992), contrite-tit-for-tat (Sugden 1986, Boerlijst et al 1997) or win-stay, lose-shift (Nowak & Sigmund 1993). In general, it is a very difficult task to find successful strategies for playing the repeated PD (Axelrod 1984, Kraines & Kraines 1988, Fudenberg & Maskin 1990, Lindgren 1991).

But if we merely want to analyze whether cooperation has any chance to evolve by direct reciprocity at all, then a very simple game can be studied. We only need to consider two strategies. Defectors, D, defect all the time. Cooperators, C, start with a cooperation and then continue to cooperate as long as the opponent cooperates, but defect if the opponent defects. These ‘reciprocators’ can be thought of as playing a strategy like tit-for-tat or Grim. Tit-for-tat (TFT) cooperates on the first move and then does whatever the opponent has done on the previous move. Grim cooperates until the opponent defects once and then permanently switches to defection.

Denote by w the probability of playing another round. The average number of rounds between two individuals is given by 1/(1 − w). The payoff matrix for cooperators (reciprocators) versus defectors is given by

$$\begin{array}{l}\begin{array}{cc}C& D\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}\frac{b-c}{1-w}& -c\\ b& 0\end{array}\right)\end{array}$$ (1)

Two cooperators help each other in every round; each cooperator pays a cost c and receives a benefit b. A cooperator helps a defector only in the first round and then refuses help. Hence, a defector gets a single benefit, b, from a cooperator, who has a single cost, c. Defectors never help each other; their mutual payoff is zero.

From payoff matrix (1) it is easy to see that cooperators are evolutionarily stable versus defectors if

$$w>c/b.$$ (2)

If the probability of playing another round is greater than the cost-to-benefit ratio, then direct reciprocity allows evolution of cooperation. A similar result can be found in Trivers (1971) who ascribed it to a personal communication with Bill Hamilton.

Another mechanism for evolution of cooperation is network reciprocity (Ohtsuki et al 2006). Cooperators and defectors occupy the vertices of a graph. Each cooperator pays a cost c for each of his neighbors to receive a benefit, b. Defectors pay no costs and distribute no benefits. In such a setting, cooperators can thrive in clusters (Nowak & May 1992). A surprisingly simple condition determines if cooperators can outcompete defectors. If the benefit-to-cost ratio, b/c, is larger than the (average) number of neighbors, k, then cooperators can win:

$$b/c>k.$$ (3)

This result holds for death-birth updating and weak selection (Ohtsuki et al 2006). Death-birth updating means that a random individual is chosen to die, and the neighbors compete for the empty site proportional to their fitness.

Network reciprocity is the generalization of spatial reciprocity (Nowak & May 1992) to evolutionary graph theory (Lieberman et al 2005). There has been much interest for studying evolutionary game dynamics in spatial settings (Matsuda 1987, Vickers 1989, Hutson & Vickers 1992, Nowak & May 1992, 1993, Herz 1994, Lindgren & Nordahl 1994, Nowak et al 1994, Killingback & Doebeli 1996, Nakamaru et al 1997, 1998, Szabó & Tőke 1998, van Baalen & Rand 1998, Szabó et al 2000, Szabó & Hauert 2002, Brandt et al 2003, Hauert & Doebeli 2004, Ifti et al 2004, Nakamaru & Iwasa 2005, 2006) and on graphs (Abramson & Kuperman 2001, Hauert & Szabó 2005, Santos & Pacheco 2005, Antal et al 2006, Ohtsuki & Nowak 2006a, Santos et al 2006a,b). There is a long-standing research effort to study how population structure affects evolutionary (Maruyama 1970, Wilson 1975, Slatkin 1981, Anderson & May 1991, Barton 1993, Pastor-Satorras & Vespignani 2001, Whitlock 2003) and ecological dynamics (Levin & Paine 1974, Levin 1992, Hassell et al 1994, Durrett & Levin 1994, Durrett 1999, Neuhauser 2001, Dunne 2006, May 2006).

In this paper, we bring together direct and network reciprocity. The individuals of a population occupy the vertices of a graph and play the repeated Prisoner's Dilemma with all their neighbors. Hence, we study the evolution of direct reciprocity in a spatial setting and in social networks.

Nakamaru et al (1997, 1998) are precursors in this direction. They studied the competition between tit-for-tat (TFT) and unconditional-defector (ALLD) strategies on cycles and square lattices adopting pair approximation, pair-edge approximation, and methods of stochastic processes. They found that lattices promote cooperation; in particular, TFT can invade an ALLD-population. In addition, they studied two different updating rules, called the score-dependent fertility model and the score-dependent viability model, respectively, and found that the former is more favorable for cooperation than the latter.

Here we will derive the fundamental conditions for the evolution of cooperation in a model where direct reciprocity and network reciprocity occur simultaneously. We study both infinite and finite populations which structure is described by a regular graph. We study four different updating mechanisms and provide analytic derivations for all results. Our calculation is based on weak selection.

In Section 2 we make preliminary remarks about evolutionary game dynamics in finite and infinite populations and about evolutionary graph theory. In Section 3 we discuss direct reciprocity in a well-mixed population. In Section 4 we discuss direct reciprocity on graphs. In Section 5, we introduce the idea of an error rate that increases with the number of interacting partners. Section 6 is the conclusion.

2. Preliminary remarks

2.1 Evolutionary stability, risk-dominance and the 1/3-rule

Evolutionary game theory studies frequency dependent selection; the fitness of individuals is not constant, but depends on the composition of the population (Maynard Smith 1982, Hofbauer & Sigmund 2003, Nowak & Sigmund 2004). Consider a game between two strategies S₁ and S₂ given by the payoff matrix

$$\begin{array}{l}\begin{array}{cc}{S}_1& S_2\end{array}\\ \begin{array}{c}{S}_1\\ S_2\end{array}\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\end{array}$$ (4)

The quantities a_ij denote the payoff for strategy S_i versus S_j.

Strategy S₁ is a strict Nash equilibrium (or evolutionarily stable) if a₁₁ > a₂₁. This means that an infinitely large population of S₁ players cannot be invaded by S₂ under deterministic selection dynamics.

In an infinite population, if a₁₁ > a₂₁ and a₁₂ < a₂₂ then both strategies are evolutionarily stable. The replicator equation (Taylor & Jonker 1978, Hofbauer et al 1979, Weibull 1995, Hofbauer & Sigmund 1998) admits an unstable mixed equilibrium. Strategy S₁ is risk-dominant (RD) if it has the bigger basin of attraction. This is the case if a₁₁ + a₁₂ > a₂₁ + a₂₂.

In a population of finite size, the crucial quantity is the fixation probability of a strategy, defined as the probability that the lineage arising from a single mutant of that strategy will take over the entire population consisting of the other strategy (Nowak et al 2004, Taylor et al 2004). If

$$a_{11}+2a_{12}>a_{21}+2a_{22}$$ (5)

then the fixation probability of strategy S₁ is greater than the inverse of the population size, 1/N. This means selection favors the replacement of S₂ by S₁, and therefore a single S₁ player in a population of S₂ players is an advantageous mutant. The condition can be expressed as a 1/3 rule: if the fitness of the invading S₁ at a frequency of 1/3 is greater than the fitness of the resident S₂ then the fixation probability of S₁ is greater than 1/N (Nowak et al 2004, Imhof & Nowak 2006). This condition holds in the limit of weak selection.

Note also: if S₁ is risk-dominant (RD) when compared to S₂, which means that a₁₁ + a₁₂ > a₂₁ + a₂₂, then the fixation probability of S₁ is greater than the fixation probability of S₂ for weak selection and large population size (Nowak et al 2004, Imhof & Nowak 2006).

2.2 Games on graphs

Imagine the individuals of a population arranged on the vertices of a graph. Each vertex is occupied by one individual which plays a particular strategy. There are no empty vertices. The edges determine who interacts with whom. Each individual obtains its payoff by playing the game with all connected individuals. There can be a number of different update mechanisms, which specify how the composition of the population changes under natural selection.

Birth-death (BD) updating means that at any one time step an individual is selected for reproduction proportional to fitness; the offspring replaces a randomly-chosen neighbor. For pairwise comparison (PC) updating a randomly chosen player compares payoffs with his random neighbor; a player adopts the strategy of the other player with a certain probability depending on payoff (Traulsen et al 2006). Death-birth (DB) updating means that a random individual is selected to die (or to update its strategy) and the neighbors compete for the empty site proportional to their fitness (we note that DB updating corresponds to the score-dependent fertility model by Nakamaru et al. (1998)). Imitation (IM) updating means that a randomly chosen player either stays with his own strategy or adopts one of the neighbors' strategies proportional to fitness.

Each of these update rules specifies a stochastic process which defines how the composition of the population (that is the coloring of the graph) changes over time. For weak selection and large population size, the evolutionary game dynamics on regular graphs can be studied by a deterministic replicator equation with a transformed payoff matrix (Ohtsuki & Nowak 2006b). This ‘replicator equation on graphs’ describes how the expected frequency of each strategy changes over time. For a game between two strategies, the transformed payoff matrix is given by

$$\begin{array}{l}\begin{array}{cc}{S}_1& S_2\end{array}\\ \begin{array}{c}{S}_1\\ S_2\end{array}\left(\begin{array}{cc}{a}_{11}& a_{12}+H\\ a_{21}-H& a_{22}\end{array}\right)\end{array}$$ (6)

The term, H, depends on the update rule. For BD and PC updating, we have

$$H=\frac{a_{11}+a_{12}-a_{21}-a_{22}}{k-2}.$$ (7)

For DB updating, we have

$$H=\frac{\left(k+1\right)a_{11}+a_{12}-a_{21}-\left(k+1\right)a_{22}}{\left(k+1\right)\left(k-2\right)}.$$ (8)

For IM updating, we have

$$H=\frac{\left(k+3\right)a_{11}+3a_{12}-3a_{21}-\left(k+3\right)a_{22}}{\left(k+3\right)\left(k-2\right)}.$$ (9)

The parameter k denotes the degree of the graph, which is the number of neighbors of each individual. Note that all three expressions of H converge to 0 as k → ∞; therefore, the normal payoff matrix (and hence the standard replicator equation) is obtained in the limit where every individual interacts with every other individual, which is the case of the complete graph.

The calculation is based on pair-approximation for regular graphs with degree k ≥ 3 and can be found in Ohtsuki & Nowak (2006b). Note that BD and PC updating favors the risk dominant strategy, while DB and IM updating favors strategy S₁ if a₁₁ is large and a₂₂ is small. We can use this formalism to study any game on any regular graph. In this paper, we investigate the repeated Prisoner's Dilemma.

3. Direct reciprocity in a well-mixed population

Let us now use the formalism of Section 2.1 to study direct reciprocity in a well-mixed population, where every individual interacts equally likely with every other individual. Recall from the introduction that the characteristic payoff matrix of direct reciprocity is of the form

$$\begin{array}{l}\begin{array}{cc}C& D\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}\frac{b-c}{1-w}& -c\\ b& 0\end{array}\right)\end{array}$$ (10)

Thus it is easy to see that cooperators are evolutionarily stable if w > c/b. This condition can also be written as

$$\frac{b}{c}>\frac{1}{w}.$$ (11)

In this case, a single defector in an infinitely large population of cooperators has a lower fitness. It is still possible, however, that in a finite population the fixation probability of a single defector, ρ_D, is greater than the inverse of the population size, 1/N.

If we want defectors to be disadvantageous, we must ask that ρ_D < 1/N. For weak selection and large population size this is the case if

$$\frac{b}{c}>\frac{3-w}{2w}.$$ (12)

This condition ensures that the basin of attraction of cooperators is greater than 1/3.

Cooperators have a larger basin of attraction than defectors (they are risk-dominant) if

$$\frac{b}{c}>\frac{2-w}{w}.$$ (13)

In this case, we have ρ_C > ρ_D for weak selection and large population size.

For cooperators to be advantageous, we must ask that ρ_C > 1/N. For weak selection and large population size this is the case if

$$\frac{b}{c}>\frac{3-2w}{w}.$$ (14)

This condition ensures that the basin of attraction of cooperators is greater than 2/3. Note that inequalities (11) to (14) demand increasing benefit-to-cost ratios. Only if inequality (14) holds is the fixation probability of a single cooperator in a population of defectors greater than that of a neutral mutant (which is 1/N).

4. Direct reciprocity and network reciprocity

Let us now study direct reciprocity on graphs. We use the formalism outlined in Section 2.2 to construct the relevant payoff matrix, and then we use the formalism of Section 2.1.

4.1 Pairwise comparison or birth-death updating

If we use birth-death (BD) updating or pairwise comparison (PC), then direct reciprocity on a graph with degree k can be described by the transformed payoff matrix

$$\begin{array}{l}\begin{array}{cc}C& D\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}\frac{b-c}{1-w}& -c+H\\ b-H& 0\end{array}\right)\end{array}$$ (15)

where

$$H=\frac{\left(b+c\right)w-2c}{\left(1-w\right)\left(k-2\right)}.$$ (16)

From payoff matrix (15) we obtain that cooperators are evolutionarily stable if

$$\frac{b}{c}>\frac{1}{w}\frac{k-w}{k-1}.$$ (17)

Evolutionary stability on a graph means that not a single mutant but a small fraction of rare mutants (in the proper configuration) cannot spread. Note that condition (17) demands a larger benefit-to-cost ratio than the corresponding condition (11) for well mixed populations. Hence, the structured population makes it harder for direct reciprocity to be evolutionarily stable (for PC or BD updating). Spatial effects oppose cooperation in this setting.

We observe that the condition for defectors to be disadvantageous, ρ_D < 1/N, is also harder to fulfill on a graph with small k:

$$\frac{b}{c}>\frac{3-w}{2w}+\frac{3\left(1-w\right)}{2w\left(2k-1\right)}.$$ (18)

The condition for cooperators to be risk-dominant, however, is identical for structured and unstructured populations:

$$\frac{b}{c}>\frac{2-w}{w}.$$ (19)

Surprisingly, the condition for cooperators to be advantageous, ρ_C > 1/N, is less restrictive on a graph:

$$\frac{b}{c}>\frac{3-2w}{w}-\frac{3\left(1-w\right)}{w\left(k+1\right)}.$$ (20)

For the first time we observe an advantage caused by network reciprocity (graph selection).

Note that all conditions (17) to (20) converge to the corresponding conditions (11) to (14) for large k. A well-mixed population (which is sometimes called an unstructured population) is simply given by a complete graph where all individuals are connected to each other. Note also that the critical benefit-to-cost ratios expressed by inequalities (17) and (18) are declining functions of the degree of the graph, k, while inequality (20) specifies a critical benefit-to-cost ratio which is an increasing function of k.

The reason why the network opposes cooperators in conditions (17) and (18) and favors cooperators in condition (20) can be seen by inspecting the quantity H, given by eq. (16). Note that H > 0 if and only if b/c > (2 − w)/w. This inequality is precisely the condition for cooperators to be risk dominant. For smaller b/c ratios, H is negative and therefore the effect of the network is to oppose cooperators. For larger b/c ratios, H is positive and therefore the effect of the network is to favor cooperators. In general, pairwise comparison or birth-death updating favors the risk dominant strategy; as can be seen from eq (7).

Finally, we note that payoff matrix (15) is also compatible with cooperators to dominate defectors. This is the case if

$$\frac{b}{c}>1+\frac{k\left(1-w\right)}{w}.$$ (21)

If this condition holds, then selection favors cooperators over the entire frequency range. In well-mixed populations, direct reciprocity alone never allows cooperators to dominate defectors.

4.2 Death-birth updating

Next we study direct reciprocity on a graph using death-birth (DB) updating. The transformed payoff matrix is the same as (15), but H is different from the previous example. It is given by

$$H=\frac{\left(b+c\right)w+\left\{kb-\left(k+2\right)c\right\}}{\left(1-w\right)\left(k+1\right)\left(k-2\right)}.$$ (22)

From (22) we obtain the following results. Cooperators are evolutionarily stable if

$$\frac{b}{c}>\frac{k^2-w}{k+w(k^2-k-1)}.$$ (23)

Defectors are disadvantageous, ρ_D < 1/N, if

$$\frac{b}{c}>\frac{3-w}{2w}-\frac{3\left(3k-w\right)\left(1-w\right)}{2w\{(2k^2-2k-1)w+3k\}}.$$ (24)

Cooperators are risk-dominant over defectors if

$$\frac{b}{c}>\frac{2-w}{w}-\frac{4\left(1-w\right)}{w\left\{\left(k-1\right)w+2\right\}}.$$ (25)

Cooperators are advantageous, ρ_C > 1/N, if

$$\frac{b}{c}>\frac{3-2w}{w}-\frac{3\left(3k+w\right)\left(1-w\right)}{w\{(k^2-k+1)w+3k\}}.$$ (26)

Finally, cooperators dominate defectors if

$$\frac{b}{c}>1+\frac{k\left(k-1\right)\left(1-w\right)}{k+w}.$$ (27)

All critical benefit-to-cost ratios are increasing functions of k. Each of the conditions (23)-(27) demands a lower b/c ratio than the corresponding condition in a well-mixed population. Therefore, DB updating always facilitates the evolution of cooperation on graphs. Intuitively, this is because cooperators compete with defectors who are not always immediate neighbors of each other. The intuition behind the difference between BD and DB updating is easiest to see for games on cycles (Ohtsuki & Nowak 2006a).

In Table 1, we summarize the five conditions for well-mixed populations, as well as for BD and DB updating on graphs. In the Appendix A, we derive the five conditions for IM updating.

Table 1.

The critical b/c ratios.

	well mixed	BD	DB
C is ESS	$\frac{1}{w}$	$\frac{1}{w}·\frac{k-w}{k-1}$	$\frac{k^2-w}{k+(k^2-k-1)w}$
ρD < 1/N	$\frac{3-w}{2w}$	$\frac{3-w}{2w}+\frac{3\left(1-w\right)}{2w\left(2k-1\right)}$	$\frac{3-w}{2w}-\frac{3\left(3k-w\right)\left(1-w\right)}{2w\{(2k^2-2k-1)w+3k\}}$
C is risk-dominant	$\frac{2-w}{w}$	$\frac{2-w}{w}$	$\frac{2-w}{w}-\frac{4\left(1-w\right)}{w\left\{\left(k-1\right)w+2\right\}}$
ρC > 1/N	$\frac{3-2w}{w}$	$\frac{3-2w}{w}-\frac{3\left(1-w\right)}{w\left(k+1\right)}$	$\frac{3-2w}{w}-\frac{3\left(3k+w\right)\left(1-w\right)}{w\{(k^2-k+1)w+3k\}}$
C dominates	∞	$1+\frac{k\left(1-w\right)}{w}$	$1+\frac{k\left(k-1\right)\left(1-w\right)}{k+w}$

4.3 Numerical examples

Table 2 shows the critical benefit-to-cost ratios for cooperators to be evolutionarily stable for BD and DB updating on graphs and in a well-mixed population. For BD updating we observe that small k opposes cooperation: the ESS conditions are stricter than a well-mixed population. For DB updating, in contrast, smaller k greatly favors cooperation: the ESS conditions are easier to fulfill than a well-mixed population. The parameter k denotes the degree of the graph: the number of neighbors per individual. Our calculations are valid for k ≥ 3. The well-mixed population is the limit k → ∞.

Table 2.

The critical b/c ratios for C to be an ESS.

w	BD			DB			well-mixed
	k = 3	k = 10	k = 100	k = 3	k = 10	k = 100
0.1	14.5	11	10.09	2.54	5.29	9.18	10
0.5	2.5	2.11	2.01	1.55	1.83	1.98	2
0.9	1.17	1.12	1.11	1.08	1.10	1.11	1.11

Table 3 shows the critical benefit-to-cost ratios for cooperators to be advantageous (meaning to have a fixation probability greater than the inverse of the population size). Both BD and DB updating favor cooperation on graphs compared to a well-mixed population, but the effect is small for BD updating. For DB updating, we need either small k or large w in order to get small b/c thresholds. If w is small, then cooperation can only evolve because of network reciprocity. This case corresponds to frequent updating: players change their strategies after having a few interactions. If w is close to one, on the other hand, then network reciprocity has only a small effect.

Table 3.

The critical b/c ratios for C to be advantageous mutants.

w	BD			DB			well-mixed
	k = 3	k = 10	k = 100	k = 3	k = 10	k = 100
0.1	21.25	25.55	27.73	2.67	7.21	21.72	28
0.5	3.25	3.73	3.97	1.72	2.79	3.83	4
0.9	1.25	1.30	1.33	1.12	1.24	1.32	1.33

Table 4 shows the critical benefit-to-cost ratio for cooperators to dominate defectors. In a well-mixed population, this is never possible. For small b/c thresholds we need small k. For BD updating we need small k and large w.

Table 4.

The critical b/c ratios for C to dominate D.

w	BD			DB			well-mixed
	k = 3	k = 10	k = 100	k = 3	k = 10	k = 100
0.1	28	91	901	2.74	9.02	90.01	∞
0.5	4	11	101	1.86	5.29	50.25	∞
0.9	1.33	2.11	12.11	1.15	1.83	10.81	∞

5. Erroneous moves

Errors are always important when considering the repeated Prisoner's Dilemma (May 1987, Boyd 1989, Nowak & Sigmund 1990, Fudenberg & Harris 1992, Fudenberg & Levine 1998). Here we can study the effect of an error rate that is an increasing function of the number of different people with whom one person is interacting.

Suppose the cooperators, C, use the Tit-For-Tat (TFT) strategy, but in each round of the game they mistakenly choose the wrong move with probability s. If s is strictly positive and if the game is infinitely repeated (w → 1), then the average payoff per round is given by

$$\begin{array}{l}\begin{array}{cc}C& D\end{array}\\ \begin{array}{c}C\\ D\end{array}\left(\begin{array}{cc}\frac{b-c}{2}& -cs\\ bs& 0\end{array}\right).\end{array}$$ (28)

Note that two TFT players simply get the average payoff of the four possible outcomes, b − c, −c, b and 0. The defectors, D, use the ‘always defect’ (ALLD) strategy without making any mistakes (they do not even know how to cooperate). Hence, two defectors have 0 payoff. In the game between TFT and ALLD, TFT cooperates by mistake with probability s.

Interacting with more people simultaneously requires more memory and more cognitive abilities. Hence, we expect that the error rate s is an increasing function of k. Suppose that s is given by

$$s=\frac{k}{2\left(k+\kappa \right)}.$$ (29)

The maximum error rate is s = 1/2 as k → ∞; in this case TFT behaves like the random strategy which flips an unbiased coin for every move. The parameter κ denotes the number of interacting partners at which the error rate is s = 1/4 (which is half of its maximum value).

Table 5 shows the critical b/c ratio for cooperators to be advantageous mutants. Even for the infinitely repeated game (w → 1), we need small k in order to have reasonably small b/c thresholds. Playing on a sparse graph becomes essential.

Table 5.

The critical b/c ratios for TFT to be advantageous mutants in an infinitely repeated PD.

κ	BD			DB
	k = 3	k = 10	k = 100	k = 3	k = 10	k = 100
5	2.35	6.45	60.41	1.50	3.98	37.66
10	1.68	3.73	30.70	1.29	2.79	23.50
50	1.14	1.55	6.94	1.07	1.43	6.50

6. Discussion

Direct reciprocity is a mechanism for evolution of cooperation, based on the idea of repeated encounters between the same two individuals (Trivers 1971). Direct reciprocity requires accurate recognition of other players, an excellent memory of past moves and a not too heavily discounted future (May 1987). Complicated behavioral strategies can emerge in the framework of direct reciprocity (Axelrod 1984). Network reciprocity is based on the idea that each individual only (or predominantly) interacts with a well-defined, small subset of the population (Ohtsuki et al 2006). In this case, there is no need for complicated cognitive abilities. Unconditional cooperators can prevail over defectors. Network reciprocity is the generalization of spatial reciprocity (Nowak & May 1992) to evolutionary graph theory (Lieberman et al 2005).

In this paper, we have analyzed the interplay between direct and network reciprocity. The individuals of a population occupy the vertices of a graph. Neighbors play the repeated Prisoner's Dilemma. Thereby, all players accumulate a payoff which is translated into fitness. Our aim is to derive the critical benefit-to-cost ratio, b/c, which favors evolution of cooperation. In the limit of weak selection and large population size, this critical ratio depends only on two parameters: (i) the probability w to play another round in the repeated game; and (ii) the number k of neighbors per individual. In order to calculate the critical b/c ratio, we only need to study the interaction between two simple strategies: defectors use ‘always defect’ (ALLD), cooperators use tit-for-tat (TFT).

We have studied four update rules for the evolutionary dynamics. Birth-death (BD) and pairwise comparison (PC) updating give rise to identical results. Death-birth (DB) and imitation (IM) updating lead to results that differ qualitatively from BD updating. BD (and PC) updating makes it harder for cooperators to be evolutionarily stable, but easier to be advantageous. Cooperators are advantageous if a single cooperator starting in a random position on the graph has a fixation probability greater than the inverse of the population size. DB and IM updating make it easier for cooperators to be evolutionarily stable and advantageous. For DB and IM updating there is always a synergistic interaction between network reciprocity and direct reciprocity.

For DB and IM updating we find that we need either small k or large w in order to obtain reasonably small critical b/c ratios for the evolution of cooperation. Thus, either strong network reciprocity (small k) or strong direct reciprocity (large w) is sufficient for the evolution of cooperation. Any one of the two mechanisms is enough to ensure that cooperators (reciprocators) are favored.

On a graph (with finite k), it is possible that cooperators dominate defectors, which means that selection favors cooperators over the entire frequency range. In order to obtain small b/c ratios for cooperators to dominate defectors we need small k.

Nakamaru et al. (1997, 1998) studied the interaction between TFT and ALLD strategies on lattices. Quantitative comparison between the present study and Nakamaru et al. is difficult, as our calculation is based on weak selection while theirs is not. Qualitatively, however, our result agrees with that of Nakamaru et al.; (i) spatial structure generally facilitates cooperation, (ii) smaller k and larger w favor cooperation more, and (iii) TFT can dominate ALLD even though ALLD is an evolutionarily stable strategy in a well-mixed population.

Finally, we have explored the effect of an error rate which increases with the number of interacting partners per individual. In this case, we find again that small k is necessary for evolution of cooperation at reasonably small b/c ratios. Therefore, in the context of a realistic error rate (which is related to the social complexity of the situation), direct reciprocity is only effective if played in a well-defined social network where each individual has meaningful interactions only with a few others.

Appendix A. IM updating

For imitation (IM) updating, a randomly chosen player either stays with his own strategy or adopts one of the neighbors' strategies proportional to fitness. Ohtsuki & Nowak (2006b) has shown that the evolutionary dynamics of the game (4) for IM updating is described by a replicator dynamics with a transformed payoff matrix (6), where the ‘correction’ term H in (6) is

$$H=\frac{\left(k+3\right)a_{11}+3a_{12}-3a_{21}-\left(k+3\right)a_{22}}{\left(k+3\right)\left(k-2\right)}.$$ (A.1)

Let us study the direct reciprocity game eq. (10) played on a regular graph of degree k ≥ 3 under IM updating. The correction term H is calculated as

$$H=\frac{3\left(b+c\right)w+\left\{kb-\left(k+6\right)c\right\}}{\left(1-w\right)\left(k+3\right)\left(k-2\right)}.$$ (A.2)

From (A.2) we have the following results. Cooperators are evolutionarily stable if

$$\frac{b}{c}>\frac{k^2+2k-3w}{k+w(k^2+k-3)}.$$ (A.3)

Defectors are disadvantageous in invasion, ρ_D < 1/N, if

$$\frac{b}{c}>\frac{3-w}{2w}-\frac{9\left(k-w\right)\left(1-w\right)}{2w\{(2k^2+2k-3)w+3k\}}.$$ (A.4)

Cooperators are risk-dominant over defectors if

$$\frac{b}{c}>\frac{2-w}{w}-\frac{4\left(1-w\right)}{w\left\{\left(k+1\right)w+2\right\}}.$$ (A.5)

Cooperators are advantageous mutants, ρ_C > 1/N, if

$$\frac{b}{c}>\frac{3-2w}{w}-\frac{9\left(k+w\right)\left(1-w\right)}{w\{(k^2+k+3)w+3k\}}.$$ (A.6)

Cooperators dominate defectors if

$$\frac{b}{c}>1+\frac{k\left(k+1\right)\left(1-w\right)}{k+3w}.$$ (A.7)

All of the conditions (A.3)-(A.7) are easier to achieve than a well-mixed population.