Stochastic dynamics of invasion and fixation

Abstract

We study evolutionary game dynamics in finite populations. We analyze an evolutionary process, which we call pairwise comparison, for which we adopt the ubiquitous Fermi distribution function from statistical mechanics. The inverse temperature in this process controls the intensity of selection, leading to a unified framework for evolutionary dynamics at all intensities of selection, from random drift to imitation dynamics. We derive a simple closed formula that determines the feasibility of cooperation in finite populations, whenever cooperation is modeled in terms of any symmetric two-person game. In contrast with previous results, the present formula is valid at all intensities of selection and for any initial condition. We investigate the evolutionary dynamics of cooperators in finite populations, and study the interplay between intensity of selection and the remnants of interior fixed points in infinite populations, as a function of a given initial number of cooperators, showing how this interplay strongly affects the approach to fixation of a given trait in finite populations, leading to counterintuitive results at different intensities of selection.

More than 30 years have passed since Maynard-Smith and Price laid the foundations of evolutionary game theory [1]. In this context, the replicator dynamics equation [2–4] has inspired mathematicians, physicists, biologists, economists, psychologists, as well as political and computer scientists to investigate the evolution of traits under natural selection [5–7]. In the absence of mutations, and whenever populations are infinite, homogeneous, and well mixed (each individual is equally likely to interact with any other individual), the (mean-field) description by the deterministic replicator dynamics in which the abundance x of a strategy changes according to its relative payoff ẋ = x(π_x − 〈π〉), has led to inumerous insights.

In spite of the insights provided by deterministic replicator dynamics, it has been long recognized that populations, being finite, will exhibit stochastic effects that may play a sizable role. Such behavior is well known in physics, where stochastic effects in finite-sized systems have been studied for a long time [8,9]. Studies of stochastic evolutionary dynamics in finite populations, however, have been carried out mostly numerically. Indeed, finite-size populations are an ever-present ingredient in individual-based computer simulations [10–18], which naturally incorporate such stochastic effects. Different intensities of selection have been employed in these studies, ranging from a strong selection framework, captured by the finite-population analog of replicator dynamics [13–17], to an extreme selection pressure under imitation dynamics, used as a metaphor of cultural evolution [4].

Recently, the fixation probability of a trait under frequency-dependent selection in a finite population has been derived [19]. In order to control the interplay of drift and selection, an intensity of selection has been proposed, which measures the impact of the game on the fitness and determines the amplitude of fitness differences on which selection acts. In such a system, fitness is not the number of offspring, but a measure for the potential to reproduce. Analytical results were obtained for weak selection, when the result of the game acts as a linear perturbation to neutral drift. The biological relevance of this weak selection stems from the fact that most evolutionary changes are near neutral, i.e., have small influence on fitness [20]. Therefore, neutral selection can serve as a reference point [19,21]. It has been shown that the finiteness of populations may indeed lead to fundamental changes in the conventional picture emerging from deterministic replicator dynamics in infinite populations. For instance, a single cooperator in a prisoner's dilemma game has now a small yet nonzero probability of survival in a finite population of defectors. Conversely, it is not always certain that defectors wipe out cooperators in finite populations [22,23].

In physics, stochastic effects are often described in terms of an effective temperature, a methodology that has proven both successful and insightful at all scales. In line with this long tradition, we introduce here a temperature of selection in the evolutionary game dynamics of finite populations. This temperature controls the balance between selection and random drift in finite populations, providing a convenient framework to study evolutionary game dynamics at all intensities of selection—from neutral, random drift, up to the extreme limit of cultural imitation dynamics. We define the pairwise comparison process, which utilizes the ubiquitous Fermi distribution function at finite temperature. Making use of the Fermi distribution, we provide a closed analytical expression for the fixation probability in a finite, well-mixed population of arbitrary size, for an arbitrary number of cooperators at the start, and for an arbitrary intensity of (frequency-dependent) selection. As a result, we are able to explore the role of the temperature of selection in evolutionary game dynamics in what concerns the overall fixation probabilities for different dilemmas.

General symmetric two player games can be interpreted in terms of social dilemmas of cooperation [24–27], in which cooperators and defectors interact via the payoff matrix

(1)

In all dilemmas mutual cooperation is favored (R > P), although individual reasoning often leads to mutual defection. In the prisoner's dilemma (T > R > P > S), the only stable evolutionary equilibrium of the replicator dynamics is the situation in which all individuals are defectors. Other dilemmas of cooperation are more favorable to cooperation, though. For instance, whenever T > R > S > P, we enter the realm of the snowdrift game, where it is best to do the opposite of what the other player does: to cooperate if the other player defects and to defect if the other player cooperates [28]. This leads to the stable coexistence of cooperators and defectors under replicator dynamics, at the cooperator frequency

$$x^{\ast }=\frac{S-P}{T+S-R-P}.$$ (2)

In other words, starting from any given nonzero frequency of cooperators in the population, the system will evolve toward this coexistence equilibrium state. A third dilemma arises whenever R > T > P > S, the so-called stag-hunt game. In this case, one is better off doing whatever the other player will do, which corresponds to a coordination game favoring mutual cooperation (R). Indeed, replicator dynamics predicts the existence of an unstable interior fixed point at x* given by Eq. (2), such that for frequencies above this point the population will evolve toward 100% cooperation, the other fate being associated with startup frequencies below the unstable fixed point.

As usual in evolutionary game theory, the fitness of each individual is associated with the payoff resulting from interactions with other individuals from the population. Under strong selection, an individual with higher fitness will always replace an individual with lower fitness. Let us consider the following process: Two individuals are chosen randomly from the entire population—for example a cooperator C and a defector D, with payoffs π_C and π_D, respectively. Under imitation dynamics, the fitter replaces the less fit with probability p = 1. Similarly, under replicator dynamics selection always increases the fraction of the fitter type. However, most evolutionary changes have a small impact on fitness and sometimes the less successful individual may actually replace the more successful one. Whereas such occasional errors may be safely neglected in large populations, they can have decisive effects in finite populations [19]. These effects are nicely captured by replacing the probability p above with the Fermi function [5,6,18,19],

$$p=\frac{1}{1+e^{-\beta ({\pi }_C-{\pi }_D)}},$$

which gives the probability that C replaces D. Such a process we call pairwise comparison. The inverse temperature β ≥ 0 controls the intensity of selection here. For β → ∞, the situation described above is recovered, i.e., the probability for replacement of the less fit individual is 1. However, for any finite β, the reverse process may actually take place with a nonvanishing probability given by 1 − p. In particular, it is interesting to consider this process for $\beta \ll 1$, in which the fitness has a marginal influence on replacement. For $\beta \ll 1$, the process described above reduces exactly to the frequency-dependent Moran process under weak selection, studied in [19].

Let us consider the general payoff matrix Eq. (1). When the number of interactions per individual is very high, the payoffs will depend only on the fraction of both types in the population. If there are j cooperators and N − j defectors, they will accumulate payoffs π_C=(j − 1)R + (N − j)S and π_D=jT + (N − j − 1)P, respectively [29]. Self interactions are excluded. Under pairwise comparison, and in the absence of mutations, only when the two individuals chosen have different strategies can the total number of individuals with a given strategy change by 1. Similarly to the Moran process described in [19], this defines a finite state Markov process with an associated tridiagonal transition matrix. The probability to increase the number of cooperators from j to j + 1, $T_j^{+}$, and the probability to decrease that number from j to j − 1, $T_j^{-}$, are

$$T_j^{\pm }=\frac{j}{N}\frac{N-j}{N}\frac{1}{1+e^{\mp \beta ({\pi }_C-{\pi }_D)}}.$$ (3)

For large populations, this process can be approximated by a stochastic differential equation with drift $T_j^{+}-T_j^{-}$ and diffusion $\sqrt{(T_j^{+}+T_j^{-})∕N}$ [30]. For the pairwise comparison process, this yields

$$\stackrel{.}{x}=x(1-x)\text{tanh}\left(\frac{\beta }{2}({\pi }_C-{\pi }_D)\right)+\sqrt{\frac{x(1-x)}{N}}\xi$$ (4)

where x = i/N is the fraction of cooperators and ξ is Gaussian white noise with variance 1. For N → ∞, the stochastic term vanishes. In this limit, the replicator dynamics [4] is recovered from the first-order term of a high-temperature expansion, $\beta \ll 1$.

In finite populations, the quantity of interest in this process is the fixation probability of cooperators, i.e., the probability to end up in the state with N cooperators given that the initial number is k. This probability depends only on the ratio ${\alpha }_j=T_j^{-}∕T_j^{+}$. For the pairwise comparison process, this ratio reduces to α_j=exp[−β(π_C−π_D)]. In terms of α_j, the fixation probability ϕ_k is given by [31]

$${\varphi }_k=\frac{\sum _{i=0}^{k-1}\prod _{j=1}^i{\alpha }_j}{\sum _{i=0}^{N-1}\prod _{j=1}^i{\alpha }_j},$$ (5)

and the payoff difference can be written as π_C−π_D=2uj + 2v, where 2u=R − S − T + P and 2v =−R + SN − PN + P. This simplifies the fixation probability to

$${\varphi }_k=\frac{\sum _{i=0}^{k-1}\text{exp}[-\beta i(i+1)u-2\beta iv]}{\sum _{i=0}^{N-1}\text{exp}[-\beta i(i+1)u-2\beta iv]}.$$ (6)

One can interpret the sums in the previous equation as numerical approximations to the integral ${\int }_{-1∕2}^{k-1∕2}{\Pi }_{j=1}^i{\alpha }_{j}di$. The error of this approximation is of order N⁻² and will therefore rapidly vanish with increasing N. Replacing the sums by integrals leads to the surprisingly simple formula

$${\varphi }_k=\frac{\text{erf}\left[{\xi }_k\right]-\text{erf}\left[{\xi }_0\right]}{\text{erf}\left[{\xi }_N\right]-\text{erf}\left[{\xi }_0\right]}$$ (7)

valid for u ≠ 0 where $\text{erf}\left(x\right)=\frac{2}{\sqrt{\pi }}{\int }_0^{x}dy{e}^{-y^2}$ is the error function and ${\xi }_k=\sqrt{\frac{\beta }{u}}(ku+v)$. For u=0, which is in game theory sometimes referred to as “equal gains from shifting,” we find instead

$${\varphi }_k=\frac{e^{-2\beta vk}-1}{e^{-2\beta vN}-1}.$$ (8)

This result is formally identical to the fixation probability of k individuals with fixed relative fitness r=e^2βv [32,33]. Eqs. (7) and (8) reduce trivially to the neutral selection result ϕ_k=k/N for β → 0. In contrast with the results from Ref. [19], these closed expressions are valid for any intensity of selection and for any given initial number of individuals of a given trait. Two limiting cases follow immediately. $\beta \ll 1$ corresponds to weak selection and yields the generalization of the weak selection result for the birth-death process given in [19] to an arbitrary initial number of mutants. Strong selection is described by $\beta \gg 1$ and reduces the process to a semideterministic imitation process: The speed of this process remains stochastic, but the direction always increases individual fitness for β → ∞. This limit, which corresponds to the cultural evolution models of imitation dynamics, is outside the realm of the birth-death process studied in [19] and results from the nonlinearity of the Fermi distribution function.

In the following we study the evolutionary game dynamics of finite populations in some concrete cases, in connection with the results shown in Fig. 1.

FIG. 1.

(Color online) Fixation probabilities for the three qualitatively different dilemmas of cooperation describable by 2×2 games. Simulation results (filled symbols) coincide perfectly with the theoretical results Eqs. (7) and (8)(solid lines). For comparison, the theoretical result for the Moran process at maximum selection intensity (w = 1, open circles) and the neutral selection result (β = 0, diagonal line) are shown. (a) In the prisoner's dilemma, defection is dominant. As a result, only for very weak selection (β = 0.01) do cooperators acquire significant chances of fixation. (b) The stag-hunt game has an unstable equilibrium at k* = (N + 2)/2, and the transition across this point depends sensitively on the intensity of selection. (c) The snowdrift game exhibits an internal equilibrium at k* = (N − 2)/2, which is an inflection point in the theoretical curves. A small population size of N = 20 has been chosen on purpose to stress the quality of the theoretical result related to Eqs. (7) and (8). Each simulation result corresponds to the fraction of fixation of cooperators in 10⁴ independent realizations.

For the prisoner's dilemma we choose R = 3, S = 1, T = 4, and P = 2. Since u = 0, the fixation probability is given by Eq. (8). In Fig. 1(a) we show a comparison between the analytical results and extensive computer simulations for a small population of N = 20 individuals. Clearly, cooperators are always in disadvantage with respect to defectors. In spite of the exponential increase of the fixation probability of cooperators with the initial number k, only for very weak selection (β = 0.01) do cooperators acquire reasonable chances in a population as small as N = 20. As stated before, the error of the approximation leading to Eqs. (7) and (8) is of order N⁻². However, even for N = 20 excellent agreement with numerical simulations is obtained as illustrated in Fig. 1.

As a second numerical example, we consider the stag-hunt game given by the ranking of the payoff matrix illustrated in Fig. 1(b) [note that we only changed the order of numerical values from Fig. 1(a)]. This system exhibits two pure Nash equilibria, i.e., each strategy is the best reply to itself, and any selection mechanism leads to a bistable outcome. The replicator dynamics predicts a sharp transition from full defection to full cooperation at the unstable fixed point x* given by Eq. (2). In finite populations, this point is given by k* = Nx* + (P − R)/(T + S − R − P), where the second term resulting from the exclusion of self-interactions is irrelevant for large N. The transition at k* depends sensitively on the intensity of selection, becoming sharper for larger intensity of selection. For the Moran process discussed in [19] a smooth transition from zero to one is always observed in all dilemmas, even when the intensity of selection is maximized (w = 1), for which the fitness equals the payoff resulting from the game (open circles in Fig. 1). Hence, only for very smooth transitions between the two absorbing states do the predictions in [19] apply. Equation (7), instead, provides the overall scenario for all intensities of selection without need to invoke the so-called 1/3 rule.

The snowdrift game [28] (also known as the hawk-dove game or chicken) characterized by the payoff ranking T > R > S > P presents the most interesting scenario in this context. This system exhibits a mixed Nash equilibrium at k*. The replicator dynamics predicts that any initial condition will lead to this mixed equilibrium. This is clearly not possible in a finite population, in which the system ends up in one of the absorbing states k = 0 or k = N. For weak selection, the fixation probability increases smoothly with k from zero to one [open circles in Fig. 1(c)]. As β increases, a plateau develops in the center region from 25% up to 75% of cooperators, in which the fixation probability is roughly independent of k. For β = 0.05 its value is ≈ 15%. For β = 0.1, the plateau widens, but more interestingly, it shifts position, so that now the fixation probability becomes ≈ 2%. This behavior is related to the location of k*, which in the finite system leads to the occurrence of an inflection point at k* in the fixation probability curves. With increasing selection pressure the probability distribution becomes sharply peaked around k*. Hence, the fixation probability of defectors rapidly approaches 1, as k* < 1/2. In this case, the remnant of the interior fixed point leads to a fast increase of the time to fixation, which may become arbitrarily long (a detailed account of the fixation times has been presented elsewhere [34]). This observation cannot be inferred from the weak selection analysis of Ref. [19] and contrasts sharply with the predictions from the replicator dynamics equation: For the stochastic system, fixation at x = 0 or x = 1 is certain, whereas the replicator dynamics predicts coexistence of cooperators and defectors.

Throughout this work, we have followed the conventional approach of evolutionary game theory, assuming that the payoff matrices are constant and independent of β. However, the intensity of selection and/or the payoff matrix may change during evolution, a feature that would lead to a co-evolutionary process that deserves further attention. One possibility is a situation in which there is a certain amplitude of variation in the payoff values during each interaction, for fixed β. To the extent that these variations can be considered random, such an effect might contribute (in a mean-field treatment) to an overall rescaling of the selection temperature, leading to a new value β′. Such a new temperature would result from replacing the distribution of fitness values among each of the two types in the population by their average values. The theory presented here may be viewed as accounting for such a mean-field description, which will become more exact the narrower the payoff distributions.

To sum up, the pairwise comparison process discussed here provides a unified framework in which one can study evolutionary dynamics in finite populations. We derived a simple formula that determines the fixation probability for a given trait under frequency-dependent selection, at all intensities of selection, and for any given number of initial individuals of that trait present in the population. In a finite population, the only stable evolutionary outcome corresponds to all individuals sharing a common trait. The probability of fixation in either of these traits, however, is a non-trivial function of the number of individuals sharing that trait at the start, depending sensitively on the intensity of selection, thereby configuring an interesting evolutionary scenario not accounted for previously. We have shown that with increasing selection pressure the topological structure that is predicted by the deterministic replicator dynamics in infinite populations plays an increasing role. Depending on the nature of the game, we find that the transition between competing extrema in coordination games exhibiting bistability gets sharper with increasing selection pressure. On the other hand, for games exhibiting a mixed equilibrium, the fingerprint of the stable mixed equilibrium reflects itself in the rate of fixation into one of the absorbing states, which may increase prohibitively with increasing selection pressure. In other words, even when fixation is certain, the time to reach it may become arbitrarily long.