The Replicator Equation on Graphs

Abstract

We study evolutionary games on graphs. Each player is represented by a vertex of the graph. The edges denote who meets whom. A player can use any one of n strategies. Players obtain a payoff from interaction with all their immediate neighbors. We consider three different update rules, called ‘birth-death’, ‘death-birth’ and ‘imitation’. A fourth update rule, ‘pairwise comparison’, is shown to be equivalent to birth-death updating in our model. We use pair-approximation to describe the evolutionary game dynamics on regular graphs of degree k. In the limit of weak selection, we can derive a differential equation which describes how the average frequency of each strategy on the graph changes over time. Remarkably, this equation is a replicator equation with a transformed payoff matrix. Therefore, moving a game from a well-mixed population (the complete graph) onto a regular graph simply results in a transformation of the payoff matrix. The new payoff matrix is the sum of the original payoff matrix plus another matrix, which describes the local competition of strategies. We discuss the application of our theory to four particular examples, the Prisoner’s Dilemma, the Snow-Drift game, a coordination game and the Rock-Scissors-Paper game.

1. Introduction

Consider an evolutionary game with n strategies, labelled i = 1, …, n. The payoff matrix, A, is an n × n matrix, whose entries, a_ij, denote the payoff for strategy i versus strategy j. The relative abundance (frequency) of each strategy is given by x_i. We have ${\sum }_{i=1}^n{x}_i=1$. The fitness of strategy i is given by $f_i={\sum }_{j=1}^n{x}_j{a}_{ij}$. For the average fitness of the population, we obtain $\phi ={\sum }_{i=1}^n{x}_i{f}_i$. The replicator equation is given by

$${\stackrel{.}{x}}_i=x_i(f_i-\phi )i=1,\dots ,n$$ (1)

This equation is one of the fundamental equations of evolutionary dynamics. It describes evolutionary game dynamics (=frequency dependent selection) in the deterministic limit of an infinitely large, well-mixed population. Stochasticity and spatial effects are ignored. ‘Well-mixed’ means that population structure is ignored; all individuals are equally likely to interact with each other.

The replicator equation is defined on the simplex S_n, which is given by the set of all points (x₁, …, x_n) with the property ${\sum }_{i=1}^n{x}_i=1$. The simplex S_n is invariant under replicator dynamics: a trajectory which begins in the simplex, never leaves the simplex. Each face of the simplex, defined by one or several startegies being absent, is invariant. The replicator equation describes pure selection dynamics. Mutation is not considered. Each corner point of the simplex is an equilibrium. If a strategy is evolutionarily stable or a strict Nash equilibrium, then the corner point of the simplex corresponding to a homogeneous population using this strategy is an asymptotically stable fixed point. There can be at most one isolated equilibrium point in the interior of the simplex. For n ≥ 4, if there is an interior equilibrium, there can also be a limit cycle or a chaotic attractor. Many more properties of this system and the relationship to Lotka-Volterra equations of ecology are descibed in the book by Hofbauer & Sigmund (1998). The replicator equation was introduced by Taylor & Jonker (1978), followed by Hofbauer et al (1979) and Zeeman (1980). Evolutionary game theory was invented by Maynard Smith & Price (1973) and Maynard Smith (1982). For recent reviews see Hofbauer & Sigmund (2003) and Nowak & Sigmund (2004). Books on game theory and evolutionary game theory include Fudenberg & Tirole (1991), Binmore (1994), Weibull (1995), Samuelson (1997), Fudenberg & Levine (1998), Hofbauer & Sigmund (1998), Gintis (2000), and Cressman (2003).

In this paper, we study evolutionary game dynamics in structured populations (Nowak & May 1992, 1993, Ellison 1993, Herz 1994, Lindgren & Nordahl 1994, Nowak et al 1994, Killingback & Doebeli 1996, Nakamaru et al 1997, 1998, Epstein 1998, Szabó & T oke 1998, Van Baalen & Rand 1998, Watts & Strogatz 1998, Eshel et al 1999, Hartvigsen et al 2000, Page et al 2000, Szabó et al 2000, Skyrms & Pemantle 2000, Abramson & Kuperman 2001, Hauert 2001, Irwin & Taylor 2001, Ebel & Bornholdt 2002, Hauert et al 2002, Szabó & Hauert 2002, Brandt et al 2003, Le Galliard et al 2003, Hauert & Szabó 2003, Hauert & Doebeli 2004, Ifti et al 2004, Szabó & Vukov 2004, Szolnoki & Szabó 2004, Eguíluz et al 2005, Hauert 2005, Nakamaru & Iwasa 2005, Santos & Pacheco 2005, Vukov & Szabó 2005, Santos et al 2006a,b). The individuals occupy the vertices of a graph; the edges of the graph determine which individuals interact with each other (Lieberman et al 2005, Ohtsuki et al 2006). We consider n strategies and the general payoff matrix A = [a_ij]. Each individual derives a payoff, P, from the interaction with all of its neighbours in the graph. The fitness of an individual is given by 1 − w +wP, where the parameter w determines the intensity of selection. The case w → 0 represents the limit of weak selection, while w = 1 denotes strong selection, where fitness equals payoff. Strong selection is a special case, because in general the fitness of an individual will not only depend on the particular game that is under consideration, but on many different factors (Nowak et al 2004). Therefore, introducing a parameter for varying the intensity of selection is an important step, which was never taken in the traditional framework of the replicator equation, because there w cancels out.

In games on graphs, the fitness of an individual is locally determined from interactions with all adjacent individuals. The traditional replicator equation (1) describes the special case of a ‘complete graph’, where all vertices are connected to each other and hence all individuals are adjacent.

We consider three different update rules for the evolutionary dynamics (Fig 1a–c), which we call ‘birth-death’ (BD), ‘death-birth’ (DB) and ‘imitation’ (IM). (i) For BD updating, an individual is selected for reproduction from the entire population proportional to fitness; the offspring of this individual replaces a randomly chosen neighbor. (ii) For DB updating, a random individual from the entire population is chosen to die; the neighbors compete for the empty site proportional to fitness. (iii) For IM updating, a random individual from the entire population is chosen to revise its strategy; it will either keep its current strategy or imitate one of the neighbors’ strategies proportional to fitness. Note that our imitation updating is different from the ‘imitation dynamics’ introduced by Weibull (1995) and Hofbauer & Sigmund (2003), which describe deterministic game dynamics in a well-mixed population, where random pairs of players compare their payoffs and possibly imitate the strategy of the other.

Figure 1.

Four different update rules are studied in this paper. (a) Birth-death (BD) updating. A player is chosen for reproduction from the entire population proportional to fitness. The offspring replaces a randomly chosen neighbor. (b) Death-birth (DB) updating. A random player is chosen to die. The neighbors compete for the empty site proportional to their fitness. (c) Imitation (IM) updating. A random player is chosen for updating his strategy. The player keeps his current strategy or imitates one of the neighbors’ strategies proportional to fitness. (d) Pairwise comparison (PC) updating. A random player is chosen for updating his strategy. One of the neighbors is chosen at random. The first player either keeps his current strategy or adopts the neighbor’s strategy with a probability that depends on the payoff difference. Random choices are shown in dark blue. Choices that are proportional to fitness are shown in red. BD and PC updating (yellow background) lead to identical evolutionary dynamics in our present analysis. DB and IM updating (light blue background) have similar behavior.

These three update rules define three slightly different stochastic processes. In each process, one elementary step involves two random choices, one of them is proportional to fitness. For BD updating the first choice is proportional to fitness, for DB and IM updating the second choice is proportional to fitness. We will find that this detail can introduce interesting differences.

In the Appendix, we also consider a fourth update rule called ‘pairwise comparison’ (PC) (Fig 1d). Here one player is chosen at random, then one of its neighbors is chosen. The first individual will adopt the strategy of the second indvidual with a probability that is given by 1/[1+exp(− wΔP)] where the payoff difference is ΔP = P₂ − P₁. Interestingly, this update rule leads to the same behavior as BD updating in our current analysis. Therefore, we do not need to consider it as an additional case.

Games on graphs are stochastic, while the replicator equation is deterministic. Recently Traulsen et al (2005, 2006a) have found that the Moran process in a well-mixed population lead to the deterministic equation that is called adjusted replicator dynamics. What we want to do in this paper is to derive a system of ordinary differential equations that describes how the expected frequency of each strategy in a game on a graph changes over time. We will use pair-approximation (Matsuda et al 1987, 1992, Van Baalen 2000) on regular graphs of degree k (Ohtsuki et al 2006). This means each individual is connected to k other individuals. Strictly speaking pair-approximation is formulated for infinitely large Bethe lattices (or Caily trees) which have no loops and no leaves. It is well known, however, that pair-approximation gives good results for random regular graphs; as the number of vertices, N, increases the probability of short loops becomes negligible. As we will point out below our calculation requires k > 2. For an analysis of k = 2 see Ohtsuki & Nowak (2006).

Let us introduce the n × n matrix B = [b_ij] for the three different update mechanisms as follows:

$$\begin{array}{rrrl}\text{BD}:& & b_{ij}=& \frac{a_{ii}+a_{ij}-a_{ji}-a_{jj}}{k-2}\\ \text{DB}:& & b_{ij}=& \frac{(k+1)a_{ii}+a_{ij}-a_{ji}-(k+1)a_{jj}}{(k+1)(k-2)}\\ \text{IM}:& & b_{ij}=& \frac{(k+3)a_{ii}+3a_{ij}-3a_{ji}-(k+3)a_{ij}}{(k+3)(k-2)}\end{array}$$ (2)

Let us further introduce the quantities

$$g_i=\sum _{i=1}^n{x}_j{b}_{ij}.$$ (3)

If x_i(t) is the expected frequency of strategy i on an infinitely large graph of degree k at time t, then our pair-approximation calculation in the limit of weak selection leads to the surprisingly simple equation

$${\stackrel{.}{x}}_i=x_i(f_i+g_i-\phi )i=1,\mathrm{..},n$$ (4)

We propose to call this equation the ‘replicator equation on graphs’. It describes how the expected frequencies of strategies on a graph of degree k > 2 change over time. The simplicity and symmetry of this equation is remarkable given the complexity of the underlying stochastic process that describes games on graphs.

The term $f_i={\sum }_{j=1}^n{x}_j{a}_{ij}$ denotes the average fitness of strategy i, as in the replicator equation, and comes from well-mixed interactions among all strategies. The additional term, g_i, characterizes the local competition among strategies. Note that the population average of the local competition term sums to zero,

$$\sum _{i=1}^n{x}_i{g}_i=0.$$ (5)

Therefore the average fitness of the population, $\phi ={\sum }_{i=1}^n{x}_i(f_i+g_i)={\sum }_{i=1}^n{x}_i{f}_i$, remains the same as in the replicator equation.

As seen in eqs (2), the term for local competition, b_ij, includes the payoff that strategy i gets from strategy i plus the payoff that strategy i gets from strategy j minus the payoff that j gets from i minus the payoff that j gets from j. The diagonal terms, a_ii and a_jj, characterize the effect of assortativeness, while the off-diagonal terms, a_ij and a_ji, characterize the effect of spite. Note that the matrix (b_ij) is antisymmetric, i.e. b_ij = − b_ji. This makes sense, because the gain of one strategy in local competitiveness is the loss of another. In particular the diagonal terms b_ii are always zero, suggesting that the payoff for one strategy playing against others using the same strategy will always be the same irrespective of population structure.

In a structured population, it is especially important which payoff players get when interacting with another player who uses the same strategy (assortativeness) and also which payoff strategies provide to others with whom they are in direct competition (spite). As in eqs (2), for BD updating the contributions from assortativeness and spite are equally strong, while for DB updating assortativeness is stronger than spite (the coefficients for assortativeness in eqs (2) have relative weight k + 1). IM updating has a balance of assortativeness and spite that is somewhere between BD and DB updating.

For a zero sum game, which can be defined by a_ii = 0 and a_ij = −a_ji for all i and j, we find that b_ij is equal to a_ij times a constant. Therfore, the graph has no consequence for the evolutionary dynamics (other than affecting the time scale). For pair approximation and weak selection, a zero sum game on a regular graph has the same evolutionary dynamics as in a well-mixed population.

Observe also as k increases the relative contribution of g_i compared to f_i decreases. In the limit k → ∞, eq (4) leads back to eq (1), the replicator equation on a highly connected graph converges to the normal replicator equation, which agrees with the result by Traulsen et al (2006a) for weak selection.

Finally, we note that the replicator equation on graphs can also be written in the form

$${\stackrel{.}{x}}_i=x_i[\sum _{j=1}^n{x}_j(a_{ij}+b_{ij})-\phi ].$$ (6)

Therefore, moving evolutionary game dynamics from a well mixed population (the complete graph) onto a regular graph of degree k is simply described by a transformation of the payoff matrix

$$[a_{ij}]\to [a_{ij}+b_{ij}].$$ (7)

Our results will be derived for degree homogeneous (=regular) graphs and weak selection, but we expect that the replicator equation on graphs is also a good approximation for many games on non-regular graphs and for higher intensity of selection. In any case, an exact understanding of the limiting scenario is a good point of departure for investigations of more complicated and more specific scenarios.

The paper is structured as follows. In Sections 2,3 and 4, we will show the pair-approximation calculations for BD, DB and IM updating, respectively. In Section 5, we will study the Prisoner’s Dilemma, and in Section 6 the Snow-drift game, where we investigate the effect of spatiality on the evolution of cooperation. In Section 7 we will study a coordination game to see the possibility whether local population structure favors efficient outcomes for groups through individual selection. In Section 8 we will study the Rock-Scissors-Paper game to investigate spatial effect on evolutionary cycles. Section 9 contains conclusions. There is a short Appendix showing the equivalence between PC and BD updating.

2. Birth-death (BD) updating

For BD updating, a player is chosen for reproduction from the entire population proportional to fitness. The offspring of this player replaces a random neighbor. In this section, we will derive the replicator equation for games on graphs with BD updating, assuming weak selection w ≪ 1.

In a well-mixed population, the probability that a player meets an i-strategist is equal to its global frequency, x_i. For games on graphs, however, this is not necessarily true. Since dispersal is limited, those who use the same strategy tend to form clusters. Therefore, we have to take into account the correlation in strategies of two adjacent players.

Let q_i|j be the conditional probability that the focal player uses strategy i given that an adjacent player uses strategy j. In other words, q_i|j is the local frequency of strategy i around strategy j. The local frequency q_i|j is expressed by the global frequencies of strategies as q_i|j = x_ij/x_j. Here x_ij denotes the global pair-frequency of i-j pairs.

Similarly one can imagine more detailed local frequencies such as q_i|jl, which represents the conditional probability that the focal player uses strategy i given that an adjacent player uses strategy j and that a two-step adjacent player uses strategy l. For analytical tractability, we will adopt the pair approximation method (Matsuda et al 1987, 1992, Van Baalen 2000), which assumes q_i|jl = q_i|j. The crucial assumption is that a two-step adjacent player does not affect the focal site directly.

We are interested in the dynamics of global and local frequencies. Because we consider weak selection, global frequencies change at a rate of order w, which is very slow. Local frequencies change at a rate of order 1. Therefore, we have a separation of two time scales.

Let us first derive local frequencies at equilibrium. While local frequencies equilibrate, we can regard global frequencies as constant. Suppose that a player is chosen for reproduction on average once per unit time. Then the dynamics of local frequencies are calculated as follows

$${\stackrel{.}{q}}_{i\mid j}=\frac{{\stackrel{.}{x}}_{ij}}{x_j}=\frac{2}{k}\left[{\delta }_{ij}+(k-1)\left(\sum _l{q}_{i\mid l}{q}_{l\mid j}\right)-k{q}_{i\mid j}\right]+O(w).$$ (8)

Here δ_ij is the Kronecker delta; δ_ij = 1 if i = j, otherwise it is 0. From eq (8) and by using the identity q_i|jx_j = q_j|ix_i equilibrium local frequencies are calculated as

$$q_{i\mid j}^{\ast }=\frac{(k-2)x_i+{\delta }_{ij}}{k-1}.$$ (9)

We see that $q_{i\mid i}^{\ast }>x_i>q_{i\mid j}^{\ast }(j\ne i)$ holds. Players using strategy i have more i-neighbors than is expected by the global frequency, while players using another strategy have less i-neighbors than is expected.

Given these local frequencies, we can derive the dynamics of global frequencies. For convenience we rewrite $q_{i\mid j}^{\ast }$ as q_i|j. We invent the term ‘(i; k₁, …, k_n)-player’ denoting a player using strategy i who has k₁ neighbors with strategy 1, …, and k_n neighbors with strategy n.

Let us now consider one elementary step of BD updating.

The number of i-strategists increases by one, when (i) an (i; k₁, …, k_n)-player is chosen for reproduction and (ii) the offspring replaces a neighbor who does not use strategy i. The first event occurs with probability

$$\left[x_i·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid i}^{k_1}\cdots q_{n\mid i}^{k_n}\right)·W_{(i;k_1,\cdots k_n)}\right]/\overline{W}.$$ (10)

Here W_{(i; k1,…kn}) denotes the fitness of an (i; k₁, … k_n)-player, which is given by

$$W_{(i;k_1,\cdots k_n)}=1-w+w·\left(\sum _l{k}_l{a}_{il}\right).$$ (11)

W̄ is the average fitness in the population. The second event occurs with probability 1 − (k_i/k).

In contrast, the number of i-strategists decreases by one when (i) an (j; k₁, …, k_n)-player (j ≠ i) is chosen for reproduction and (ii) the offspring replaces an i-player. The first event occurs with probability

$$\left[x_j·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid j}^{k_1}\cdots q_{n\mid j}^{k_n}\right)·W_{(j;k_1,\cdots k_n)}\right]/\overline{W}.$$ (12)

The second event occurs with probability k_i/k.

From these calculations we obtain the expected increment of the frequency of strategy i, denoted by E[Δx_i], in one elementary step of updating, which takes time Δt. In infinite populations stochasticity resulting from random sampling vanishes and the quantity E[Δx_i]/Δt becomes equal to ẋ_i. Thus we obtain the deterministic evolutionary dynamics

$$\begin{array}{rl}{\stackrel{.}{x}}_i=& \frac{\text{E}[\mathrm{\Delta }{x}_i]}{\mathrm{\Delta }t}\\ =& \sum _{k_1+\cdots +k_n=k}\left[x_i·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid i}^{k_1}\cdots q_{n\mid i}^{k_n}\right)·W_{(i;k_1,\cdots k_n)}\right]·\left(1-\frac{k_i}{k}\right)/\overline{W}\\ & -\sum _{\begin{array}{c}{k}_1+\cdots +k_n=k\\ j\ne i\end{array}}\left[x_j·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid j}^{k_1}\cdots q_{n\mid j}^{k_n}\right)·W_{(j;k_1,\cdots k_n)}\right]·\frac{k_i}{k}/\overline{W}\\ \approx & w\frac{{(k-2)}^2}{k-1}·x_i(f_i+g_i-\phi ).\end{array}$$ (13)

We have

$$\begin{array}{rl}{f}_i=& \sum _j{x}_j{a}_{ij}\\ \phi =& \sum _i{x}_i{f}_i=\sum _{i,j}{x}_i{x}_j{a}_{ij}\\ g_i=& \sum _j{x}_j{b}_{ij}\\ b_{ij}=& \frac{a_{ii}+a_{ij}-a_{ji}-a_{jj}}{k-2}.\end{array}$$ (14)

Neglecting the constant factor, w(k − 2)²/(k − 1), which is equivalent to a change of time scale, gives us the replicator equation on graphs,

$${\stackrel{.}{x}}_i=x_i(f_i+g_i-\phi ).$$ (15)

3. Death-birth (DB) updating

For DB updating, a random player is chosen from the entire population to die. Then the neighbors compete for the vacancy proportional to their fitness. Again, we will derive the replicator equation for games on graphs using DB updating and assuming weak selection w ≪ 1.

First we derive the steady state of the local frequencies. Direct calculation shows that the dynamics of local frequencies are exactly the same as eq (8). Hence, the local frequencies converge to

$$q_{i\mid j}=\frac{(k-2)x_i+{\delta }_{ij}}{k-1}.$$ (16)

Next we study the dynamics of global frequencies. Let us consider one elementary step of DB updating.

The number of i-strategists increases by one when (i) an (j; k₁, …, k_n)-player (j ≠ i) dies and (ii) one of its i-neighbors wins the competition for the vacancy. The first event occurs with probability

$$x_j·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid j}^{k_1}\cdots q_{n\mid j}^{k_n}\right).$$ (17)

The second event occurs with probability

$$\frac{k_i{W}_{i\mid j}}{{\sum }_l{k}_l{W}_{l\mid j}}.$$ (18)

Here W_i|j represents the fitness of an i-player one of whose neighbors is j-player, given as

$$W_{i\mid j}=1-w+w·\left(a_{ij}+\sum _l(k-1)q_{l\mid i}{a}_{il}\right).$$ (19)

In contrast, the number of i-strategists decreases by one, when (i) an (i; k₁, …, k_n)-player dies and (ii) one of its neighbors not using strategy i wins the competition for the vacancy. The first event occurs with probability

$$x_i·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid i}^{k_1}\cdots q_{n\mid i}^{k_n}\right).$$ (20)

The second event occurs with probability

$$1-\frac{k_i{W}_{i\mid i}}{{\sum }_l{k}_l{W}_{l\mid i}}.$$ (21)

From these calculations we obtain

$$\begin{array}{rl}{\stackrel{.}{x}}_i=& \sum _{\begin{array}{c}{k}_1+\cdots +k_n=k\\ j\ne i\end{array}}{x}_j·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid j}^{k_1}\cdots q_{n\mid j}^{k_n}\right)·\frac{k_i{W}_{i\mid j}}{{\sum }_l{k}_l{W}_{l\mid j}}\\ & -\sum _{k_1+\cdots +k_n=k}{x}_i·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid i}^{k_1}\cdots q_{n\mid i}^{k_n}\right)·\left(1-\frac{k_i{W}_{i\mid i}}{{\sum }_l{k}_l{W}_{l\mid i}}\right)\\ \approx & w\frac{(k+1){(k-2)}^2}{k(k-1)}·x_i(f_i+g_i-\phi ).\end{array}$$ (22)

We have

$$\begin{array}{rl}{f}_i=& \sum _j{x}_j{a}_{ij}\\ \phi =& \sum _i{x}_i{f}_i=\sum _{i,j}{x}_i{x}_j{a}_{ij}\\ g_i=& \sum _j{x}_j{b}_{ij}\\ b_{ij}=& \frac{(k+1)a_{ii}+a_{ij}-a_{ji}-(k+1)a_{jj}}{(k+1)(k-2)}.\end{array}$$ (23)

Again, neglecting the constant factor yields the replicator equation on graphs,

$${\stackrel{.}{x}}_i=x_i(f_i+g_i-\phi ).$$ (24)

4. Imitation (IM) updating

For IM updating, a random player is chosen for updating his strategy from the entire population. Then he will either keep his current strategy or imitate one of the neighbors’ strategies proportional to fitness. As before, we assume weak selection w ≪ 1.

First we derive the steady state of local frequencies, regarding global frequencies as constant. Direct calculation leads to

$${\stackrel{.}{q}}_{i\mid j}=\frac{{\stackrel{.}{x}}_{ij}}{x_j}=\frac{2}{k+1}\left[{\delta }_{ij}+(k-1)\left(\sum _l{q}_{i\mid l}{q}_{l\mid j}\right)-k{q}_{i\mid j}\right].$$ (25)

From this, we obtain the steady state of local frequencies as

$$q_{i\mid j}=\frac{(k-2)x_i+{\delta }_{ij}}{k-1}.$$ (26)

As before, let us derive the dynamics of x_i. Consider an elementary step of IM updating. The number of i-strategists increases by one when (i) an (j; k₁, …, k_n)-player (j ≠ i) is chosen for updating and (ii) he imitates one of his i-neighbors. The first event occurs with probability

$$x_j·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid j}^{k_1}\cdots q_{n\mid j}^{k_n}\right).$$ (27)

The second event occurs with probability

$$\frac{k_i{W}_{i\mid j}}{W_{(j;k_1,\cdots ,k_n)}+{\sum }_l{k}_l{W}_{l\mid j}}.$$ (28)

The number of i-strategists decreases by one, when (i) an (i; k₁, …, k_n)-player is chosen for updating and (ii) he imitates one of his neighbors not using strategy i. The first event occurs with probability

$$x_i·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid i}^{k_1}\cdots q_{n\mid i}^{k_n}\right).$$ (29)

The second event occurs with probability

$$1-\frac{W_{(i;k_1,\cdots ,k_n)}+k_i{W}_{i\mid i}}{W_{(i;k_1,\cdots ,k_n)}+{\sum }_l{k}_l{W}_{l\mid i}}.$$ (30)

From these calculations we obtain

$$\begin{array}{rl}{\stackrel{.}{x}}_i=& \sum _{\begin{array}{c}{k}_1+\cdots +k_n=k\\ j\ne i\end{array}}{x}_j·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid j}^{k_1}\cdots q_{n\mid j}^{k_n}\right)·\frac{k_i{W}_{i\mid j}}{W_{(j;k_1,\cdots ,k_n)}+{\sum }_l{k}_l{W}_{l\mid j}}\\ & -\sum _{k_1+\cdots +k_n=k}{x}_i·\left(\frac{k!}{k_1!\cdots k_n!}{q}_{1\mid i}^{k_1}\cdots q_{n\mid i}^{k_n}\right)·\left(1-\frac{W_{(i;k_1,\cdots ,k_n)}+k_i{W}_{i\mid i}}{W_{(i;k_1,\cdots ,k_n)}+{\sum }_l{k}_l{W}_{l\mid i}}\right)\\ \approx & w\frac{k(k+3){(k-2)}^2}{{(k+1)}^2(k-1)}·x_i(f_i+g_i-\phi ).\end{array}$$ (31)

We have

$$\begin{array}{rl}{f}_i=& \sum _j{x}_j{a}_{ij}\\ \phi =& \sum _i{x}_i{f}_i=\sum _{i,j}{x}_i{x}_j{a}_{ij}\\ g_i=& \sum _j{x}_j{b}_{ij}\\ b_{ij}=& \frac{(k+3)a_{ii}+3a_{ij}-3a_{ji}-(k+3)a_{jj}}{(k+3)(k-2)}.\end{array}$$ (32)

Neglecting the constant factor yields the replicator equation for games on graphs using IM updating,

$${\stackrel{.}{x}}_i=x_i(f_i+g_i-\phi ).$$ (33)

5. The Prisoner’s Dilemma

Consider a Prisoner’s Dilemma game (Rapoport & Chammah 1965, Trivers 1971, Axelrod & Hamilton 1981). A cooperator pays a cost c for his opponent to receive a benefit b. We assume b > c. A defector pays nothing. The payoff matrix of this game is given by

$$\begin{array}{c}\\ C\\ D\end{array}\left(\begin{array}{cc}C& D\\ b-c& -c\\ b& 0\end{array}\right)$$ (34)

Defection, D, dominates cooperation, C. Defection is a strict Nash equilibrium. The traditional replicator equation of a well-mixed population is given by

$$\stackrel{.}{x}=x(1-x)(-c).$$ (35)

Here x represents the frequency (=relative abundance) of cooperators in the population. Equation (35) has two fixed points: (i) at x = 1 there is an unstable equilibrium where everybody cooperates; at x = 0 there is a stable equilibrium where everybody defects. Therefore x = 0 is the global attractor of these dynamics. Hence, evolutionary game theory predicts the victory of defectors in well-mixed populations.

The game dynamics can drastically change if we consider a structured population. The replicator equation of the Prisoner’s Dilemma on a graph of degree k for the three different update rules is given by

$$\begin{array}{rrrl}\text{BD}:& & \stackrel{.}{x}=& x(1-x)\frac{k}{k-2}(-c)\\ \text{DB}:& & \stackrel{.}{x}=& x(1-x)\frac{k}{(k+1)(k-2)}(b-kc)\\ \text{IM}:& & \stackrel{.}{x}=& x(1-x)\frac{k}{(k+3)(k-2)}\{b-(k+2)c\}.\end{array}$$ (36)

For BD updating, defectors always win over cooperators as in well-mixed populations. For DB updating, however, if b/c > k, then cooperators win over defectors. Similarly, for IM updating, cooperators win over defectors if b/c > k + 2. We note that these conditions are identical to those derived by Ohtsuki et al. (2006), when analyzing the fixation probabilities of cooperators and defectors on graphs. For DB updating, natural selection favors cooperators over defectors if the benefit-to-cost ratio of the altruistic act exceeds the degree of the graph, k (which denotes the number of neighbors of any one individual). Smaller connectivity, k, favors cooperators because then clustering is easier. Interestingly, Ohtsuki et al (2006) observe that the conditions b/c > k and b/c > k +2 also hold in numerical simulations of the Prisoner’s Dilemma on degree heterogeneous (=non-regular graphs) such as random graphs and scale free networks. In this case, the parameter k denotes the average number of neighbors per individual. Therefore, we conjecture that the replicator equation on graphs (eq 4) will also extend to many non-regular graphs, but we cannot prove this at present.

DB and IM updating can also predict a couple of interesting phenomena for the general Prisoner’s Dilemma game given by the payoff matrix

$$\begin{array}{c}\\ C\\ D\end{array}\left(\begin{array}{cc}C& D\\ R& S\\ T& P\end{array}\right).$$ (37)

The game is a Prisoner’s Dilemma if T > R > P > S. As a specific example, let us consider

$$\begin{array}{c}\\ C\\ D\end{array}\left(\begin{array}{cc}C& D\\ 5& 0\\ 8& 1\end{array}\right).$$ (38)

If this game is played on a graph with degree k = 3, then the corresponding replicator dynamics for DB updating is given by

$$\stackrel{.}{x}=x(1-x)(-2x+1),$$ (39)

There is a stable equilibrium at x* = 1/2. Therefore, in this example, unconditional cooperators and defectors can coexist.

As another example consider the Prisoner’s Dilemma given by the payoff matrix

$$\begin{array}{c}\\ C\\ D\end{array}\left(\begin{array}{cc}C& D\\ 15& 0\\ 16& 8\end{array}\right).$$ (40)

The replicator equation of this game for DB updating and weak selection on a regular graph with k = 3 is given by

$$\stackrel{.}{x}=x(1-x)(7x-5).$$ (41)

There is an unstable equilibrium at x* = 5/7. Hence, the system exhibits bistability between cooperation and defection.

6. The Snow-drift game

Consider a snow-drift game. Two drivers are trapped on either side of a snowdrift in a blizzard. Cooperation means to get out of the car and shovel. Defection means to relax, remain in the car and let the other one do the work. If either one of them cooperates, then both gain the benefit of getting home, b. The cost of removing the snowdrift is c. If both drivers shovel (cooperate), then the cost for each of them is c/2. It is assumed that b > c. The payoff matrix of this game is given by

$$\begin{array}{c}\\ C\\ D\end{array}\left(\begin{array}{cc}C& D\\ b-\frac{c}{2}& b-c\\ b& 0\end{array}\right).$$ (42)

Let x denote the frequency of cooperators. The traditional replicator equation describing a well-mixed population leads to stable coexistence of cooperators and defectors at x̂ = 1 − r, where r = c/(2b − c).

For DB and IM updating on a regular graph of degree k ≥ 3, we find that the equilibrium frequency of cooperators, x*, is always greater than x̂. Furthermore, we find that x* = 1 if b/c > (k² + 1)/(2k + 2) for DB updating and if b/c > (k² + 2k + 3)/(2k + 6) for IM updating. Therefore, spatial effects (graph selection) always favors cooperators for these two update rules.

For BD updating, we find that the equilibrium frequency of cooperators is greater than in the well-mixed case, x* > x̂, if b/c > 3/2. Remarkably, this condition does not depend on the degree of the graph. (but remember that all our results are derived for k ≥ 3). In addition, for BD updating some parameter choices lead to dominance of one strategy over the other. If b/c > (k+1)/2 then x* = 1, which means that defectors become extinct. If b/c < (2k − 1)/(2k − 2) then x* = 0, which means that cooperators become extinct.

Hauert & Doebeli (2004) have studied the effect of spatial structure on the snow-drift game. One of their update rules is equivalent to our PC updating and therefore similar to BD updating in our analysis (see Appendix). Based on numerical simulations, Hauert & Doebeli (2004) make the interesting observation that spatial structure can inhibit cooperation in the snow-drift game. This finding is in qualitative agreement with our result for BD updating: if b/c < 3/2 then the equilibrium frequency of cooperators on a regular graph of (small) degree k is less than the equilibrium frequency of cooperators in a well-mixed population. A quantitative comparison is difficult, however, because Hauert & Doebeli did not study the case of weak selection. Our prediction is that for weak selection and DB or IM updating, spatial structure always favors cooperators in the snow-drift game.

7. Pareto-efficiency versus risk-dominance in a coordination game

Consider the payoff matrix

$$\begin{array}{c}\\ A\\ B\end{array}\left(\begin{array}{cc}A& B\\ a& b\\ c& d\end{array}\right).$$ (43)

If a > c and d > b then both strategies A and B are strict Nash equilibria. In this case, the game is called a ‘coordination game’. It is best to do the same as the opponent; hence, both players want to coordinate their actions. But should they play A or B? If a + b < c + d, then strategy B is called risk-dominant (Harsanyi & Selten 1988). In the standard replicator equation describing a well-mixed population, the basin of attraction of B is then greater than 1/2. It could be, however, that a > d, in which case strategy A is called pareto-efficient. For both players, the best outcome is that both choose strategy A, but the risk of receiving a low payoff is minimized by choosing strategy B. This is an interesting dilemma. How does population structure affect the evolutionary dynamics of such a game?

Let us consider the specific coordination game given by the payoff matrix

$$\begin{array}{c}\\ A\\ B\end{array}\left(\begin{array}{cc}A& B\\ a& 0\\ 1& 2\end{array}\right).$$ (44)

Let us assume that the parameter a satisfies 1 < a < 3. Therefore, both strategies A and B are strict Nash equilibria, but B is always risk dominant over A. If a < 2 then B is both risk-dominant and Pareto efficient. If, however, a > 2 then an interesting conflict arises, because strategy A is pareto-efficient, while strategy B is risk-dominant.

First we study the replicator dynamics of this game in a well-mixed population. Let x denote the frequency of strategy A. There is an unstable equilibrium at x* = 2/(1 + a). As illustrated in Fig 2a, the system is bistable: if the initial fraction of A is greater than x*, then strategy A will take over the whole population; if the initial fraction of A is less than x*, then strategy B will take over the whole population. As we see in Fig 2a, strategy B always has the larger basin of attraction.

Figure 2.

Replicator dynamics of a coordination game in a well-mixed population (a) or on a regular graph of degree k = 3, 4 or 5 for three different update rules (b–d). The payoff matrix for the two strategies A and B is given by eq (44). Both strategies are strict Nash equilibria. The horizontal axes represent the parameter a. For 1 < a < 2, strategy B is both risk-dominant and Pareto efficient. For 2 < a < 3, strategy A is Pareto efficient, while strategy B is still risk-dominant. The solid line in each figure shows the boundary between the two basins of attraction. The broken line indicates the point where both basins are equally large (1/2). (a) In a well-mixed population, strategy B always has the larger basin of attraction. (b) For BD updating, the basin of attraction of strategy B is even larger than in a well-mixed population. BD updating favors risk-dominance. (c,d) For DB and IM updating, if a is close to 3, then strategy A has the larger basin of attraction. Hence, DB and IM updating can favor pareto-efficiency over risk dominance.

Let us now consider this coordination game on a graph. For BD updating, the basin of attraction of strategy B is always larger than in a well-mixed population. Therefore, BD updating favors risk dominance. For DB updating, if a > (3k + 1)/(k + 1) then strategy A has the larger basin of attraction. For IM updating, the equivalent condition is a > (3k + 7)/(k + 3). Since k ≥ 3 both conditions imply that a > 2, which means that A is Pareto efficient. Therefore, DB and IM updating of game dynamics on graphs can favor Pareto efficiency over risk dominance (Fig 2). See Ohtsuki & Nowak (2006) for similar results on the cycle (k = 2).

8. The Rock-Scissors-Paper game

Let us consider the rock-scissors-paper game (Hofbauer & Sigmund 1998). This game has three pure strategies, R₁, R₂ and R₃. In a pairwise matching, R₁ is defeated by R₂, R₂ is defeated by R₃, and R₃ is defeated by R₁. As an example, we study the rock-scissors-paper game with the payoff matrix

$$\begin{array}{c}\\ R_1\\ R_2\\ R_3\end{array}\left(\begin{array}{ccc}{R}_1& R_2& R_3\\ 0& 1& 4\\ 1& 4& 0\\ -1& 6& 2\end{array}\right).$$ (45)

Figure 3a shows the phase portrait of the replicator equation of this game in a well-mixed population. Each vertex of the simplex is an unstable equilbrium corresponding to a monomorphic population. There is an unstable equilibirum in the interior of the simplex. The Jacobian matrix at this internal equilibrium has three eigenvalues, one of them is associated with the transversal direction for the simplex S₃ and is of no consequence. The other two eigenvalues form a pair of complex conjugates and determine the stability of the equilibrium. For matrix (45), the real part of those two eigenvalues is given by Re[λ] = 1/28 > 0. The fact that this quantity is positive implies that the internal equilibrium is unstable. All orbits starting from the interior of the simplex ultimately converge to the heteroclinic cycle at the boundary, which consists of three edges, e₁ → e₂, e₂ → e₃, and e₃ → e₁. There are oscillations of increasing amplitude, which will eventually result in the extinction of two of the three strategies (see May & Leonard 1975).

Figure 3.

The replicator dynamics of the rock-scissors-paper game (eq 45) for a well-mixed population (a), or played on graphs with degree k = 3 for BD, DB and IM updating (b–d). Each panel shows the simplex S₃. Each corner point, e_i, corresponds to the monomorphic population where only strategy R_i is present. Open and solid circles in figures represent unstable and stable equilibria respectively. For the well-mixed population (a) and for BD updating (b), the internal equilibrium is unstable; all orbits converge to the heteroclinic cycle at the boundary. But for DB updating (c) and IM updating (d), the internal equilibrium is stable and becomes the global attractor of the dynamics.

Playing the rock-scissors-paper game on a graph not only changes the position of the internal equilibrium, but can also affect its stability. Figures 3b–d show the phase portraits of the replicator equation on a graph of degree k = 3 for BD (b), DB (c) and IM (d) updating, respectively. The real part of the two essential eigenvalues of Jacobian matrix at the internal equilibrium is

$$\begin{array}{rrrl}\text{BD}:& & Re[\lambda ]=& \frac{1}{30}\\ \text{DB}:& & Re[\lambda ]=& -\frac{2}{39}<0\\ \text{IM}:& & Re[\lambda ]=& -\frac{12}{475}<0.\end{array}$$ (46)

For DB and IM updating, this suggests that the internal equilibirum is stable and hence is the global attractor of the dynamics. We observe that DB updating stabilizes the internal equilbrium more than IM updating. In contrast, BD updating does not change the stability of the internal equilibrium in this example.

9. Discussion

Evolutionary game dynamics in a well-mixed population can be described by the replicator equation,

$${\stackrel{.}{x}}_i=x_i[\sum _{j=1}^n{x}_j{a}_{ij}-\phi ].$$ (47)

Here x_i denotes the frequency of strategy i, the quantities a_ij denote the payoff for strategy i versus strategy j and φ = Σ_ij a_ijx_ix_j is the average payoff in the population.

Evolutionary game dynamics on a regular graph of degree k in the limit of weak selection (w ≪ 1) can be described by the ‘replicator equation on graphs’,

$${\stackrel{.}{x}}_i=x_i[\sum _{j=1}^n{x}_j(a_{ij}+b_{ij})-\phi ].$$ (48)

For the three different update rules, birth-death (BD), death-birth (DB) and imitation (IM), the coefficients of the B matrix are given by

$$\begin{array}{rrrl}\text{BD}:& & b_{ij}=& \frac{a_{ii}+a_{ij}-a_{ji}-a_{jj}}{k-2}\\ \text{DB}:& & b_{ij}=& \frac{(k+1)a_{ii}+a_{ij}-a_{ji}-(k+1)a_{jj}}{(k+1)(k-2)}\\ \text{IM}:& & b_{ij}=& \frac{(k+3)a_{ii}+3a_{ij}-3a_{ji}-(k+3)a_{jj}}{(k+3)(k-2)}\end{array}$$ (49)

Therefore, moving a game from a well-mixed population onto a regular graph preserves the structure of the replicator equation and only results in a transformation of the payoff matrix

$$[a_{ij}]\to [a_{ij}+b_{ij}].$$ (50)

Appendix: Pairwise comparison (PC) updating

For PC updating, a random individual is chosen for updating its strategy. Then it chooses a random neighbor. The first player adopts the neighbor’s strategy with probability 1/(1 + exp[− wΔP]) where the payoff difference is ΔP = P₂ − P₁. Here w works as inverse temperature in statistical physics (Szabó & T oke 1998, Hauert & Szabó 2005, Traulsen et al 2006b,c). Unlike the three updating rules in the main text, w can be any non-negative real number here. As w → ∞, PC updating becomes deterministic: an updating player always imitates the neighbor with a higher payoff but never imitates the neighbor with a lower score. This is called imitate the better rule (Hofbauer & Sigmund 2003). In contrast, here we assume weak selection w ≪ 1.

First we derive the steady state of local frequencies, regarding global frequencies as constant. We obtain

$${\stackrel{.}{q}}_{i\mid j}=\frac{{\stackrel{.}{x}}_{ij}}{x_j}=\frac{1}{k}\left[{\delta }_{ij}+(k-1)\left(\sum _l{q}_{i\mid l}{q}_{l\mid j}\right)-k{q}_{i\mid j}\right].$$ (A.1)

From this, we obtain

$$q_{i\mid j}=\frac{(k-2)x_i+{\delta }_{ij}}{k-1}.$$ (A.2)

Let us derive the dynamics of x_i. Consider one elementary step of PC updating. The number of i-strategists increases by one, when a j-player is chosen for adopting the strategy of an i-neighbor (where j ≠ i). This event occurs with probability

$$x_j·q_{i\mid j}·{\left(1+exp\left[-(W_{i\mid j}-W_{j\mid i})\right]\right)}^{-1}.$$ (A.3)

On the other hand, the number of i-strategists decreases by one, when an i-player is chosen to adopt the strategy of a j neighbor (where j ≠ i). This event occurs with probability

$$x_i·q_{j\mid i}·{\left(1+exp\left[-(W_{j\mid i}-W_{i\mid j})\right]\right)}^{-1}.$$ (A.4)

From these calculations we obtain

$$\begin{array}{rl}{\stackrel{.}{x}}_i=& \sum _{j\ne i}{x}_j·q_{i\mid j}·{\left(1+exp\left[-(W_{i\mid j}-W_{j\mid i})\right]\right)}^{-1}\\ & -\sum _{j\ne i}{x}_i·q_{j\mid i}·{\left(1+exp\left[-(W_{j\mid i}-W_{i\mid j})\right]\right)}^{-1}\\ \approx & w\frac{{(k-2)}^2}{2(k-1)}·x_i(f_i+g_i-\phi ).\end{array}$$ (A.5)

We have

$$\begin{array}{rl}{f}_i=& \sum _j{x}_j{a}_{ij}\\ \phi =& \sum _i{x}_i{f}_i=\sum _{i,j}{x}_i{x}_j{a}_{ij}\\ g_i=& \sum _j{x}_j{b}_{ij}\\ b_{ij}=& \frac{a_{ii}+a_{ij}-a_{ji}-a_{jj}}{k-2}.\end{array}$$ (A.6)

Neglecting the constant factor yields the replicator equation for games on graphs,

$${\stackrel{.}{x}}_i=x_i(f_i+g_i-\phi ).$$ (A.7)

Note that this equation is exactly the same as for BD updating.