Evolutionary games in the multiverse

Abstract

Evolutionary game dynamics of two players with two strategies has been studied in great detail. These games have been used to model many biologically relevant scenarios, ranging from social dilemmas in mammals to microbial diversity. Some of these games may, in fact, take place between a number of individuals and not just between two. Here we address one-shot games with multiple players. As long as we have only two strategies, many results from two-player games can be generalized to multiple players. For games with multiple players and more than two strategies, we show that statements derived for pairwise interactions no longer hold. For two-player games with any number of strategies there can be at most one isolated internal equilibrium. For any number of players with any number of strategies , there can be at most isolated internal equilibria. Multiplayer games show a great dynamical complexity that cannot be captured based on pairwise interactions. Our results hold for any game and can easily be applied to specific cases, such as public goods games or multiplayer stag hunts.

Game theory was developed in economics to describe social interactions, but it took the genius of John Maynard Smith and George Price to transfer this idea to biology and develop evolutionary game theory (1–3). Numerous books and articles have been written since. Typically, they begin with an introduction about evolutionary game theory and go on to describe the Prisoner's Dilemma, which is one of the most intriguing games because rational individual decisions lead to a deviation from the social optimum. In an evolutionary setting, the average welfare of the population decreases, because defection is selected over cooperation. How can a strategy spread that decreases the fitness of an actor but increases the fitness of its interaction partner? Various ways to solve such social dilemmas have been proposed (4, 5). In the multiplayer version of the Prisoner's Dilemma, the public goods game, a number of players take part by contributing to a common pot. Interest is added to it and then the amount is split equally among all, regardless of whether they have contributed or not. Because only a fraction of one's own investment goes back to each player, there is no incentive to deposit anything. Instead, it is tempting only to take the profits of the investments of others. This scenario has been analyzed in a variety of contexts (6, 7). The evolutionary dynamics of more general multiplayer games has received considerably less attention, and we can guess why from the way William Donald Hamilton put it: “The theory of many-person games may seem to stand to that of two-person games in the relation of sea-sickness to a headache” (8). Only recently, this topic has attracted renewed interest (9–14).

As shown by Broom et al. (9), the most general form of multiplayer games, a straightforward generalization of the payoff matrix concept, leads to a significant increase in the complexity of the evolutionary dynamics. Although the evolution of cooperation is an important and illustrative example, typically it does not lead to very complex dynamics. On the other hand, intuitive explanations for more general games are less straightforward, but only they illustrate the full dynamical complexity of multiplayer games (9).

To approach this complexity, we discuss evolutionary dynamics in finite as well as infinite populations. For finite populations, we base our analysis on a variant of the Moran process (15), but under weak selection our approach is valid for a much wider range of evolutionary processes (see next section). We begin by recalling the well-studied two-player two-strategy scenario. Then, we increase the number of players, which results in a change in the dynamics and some basic properties of the games. For infinitely large populations, we explore the dynamics of multiplayer games with multiple strategies and illustrate that this new domain is very different as compared to the two-player situation (see also ref. 9). We provide some general results for these multiplayer games with multiple strategies. The two-strategy case and the two-player scenario are then a special case, a small part of a larger and more complex multiverse.

Model and Results

Two-player games with two strategies have been studied in detail, under different dynamics and for infinite as well as for finite population sizes. Typically, two players meet, interact, and obtain a payoff. The payoff is then the basis for their reproductive success and hence for the change in the composition of the population (2). This framework can be used for biological systems, where strategies spread by genetic reproduction, and for social systems, where strategies spread by cultural imitation.

Consider two strategies, A and B. We define the payoffs by α_i, where α is the strategy of the focal individual and the subscript i is the number of remaining players playing A. For example, when an A strategist meets another person playing A she gets a₁. She gets a₀ when she meets a B strategist. This leads to the payoff matrix

Some of the important properties of two-player games are:

1. Internal equilibria. When A is the best reply to B (a₀ > b₀) and B is the best reply to A (b₁ > a₁), the replicator dynamics predicts a stable coexistence of both strategies. Similarly, when both strategies are best replies to themselves, there is an unstable coexistence equilibrium. A two-player game with two strategies can have at most one such internal equilibrium.

2. Comparison of strategies. In a finite population, strategy A will replace B with a higher probability than vice versa if Na₀ + (N – 2)a₁ > (N – 2)b₀ + Nb₁. This result holds for the deterministic evolutionary dynamics discussed by Kandori et al. (16), for the Moran process and a wide range of related birth-death processes under weak selection (15, 17), and for some special processes for any intensity of selection (17). However, Fudenberg et al. (18) obtain a slightly different result for an alternative variant of the Moran process under nonweak selection. For large populations, the condition above reduces to risk dominance of A, a₁ + a₀ > b₁ + b₀.

3. Comparison with neutrality. For weak selection, the fixation probability of strategy A in a finite population is larger than neutral (1/N) if (2N – 1)a₀ + (N – 2)a₁ > (2N – 4)b₀ + (N + 1)b₁. For a large N, this means that A has a higher fitness than B at frequency 1/3, termed the one-third law (19–21). The 1/3 law holds under weak selection for any process within the domain of Kingman's coalescence (22).

Often interactions are not between two players but between whole groups of players. Quorum sensing, public transportation systems, and climate preservation represent examples of systems in which large groups of agents interact simultaneously. Starting with the seminal work of Gordon and Hardin on the tragedy of the commons (23, 24), such multiplayer games have been analyzed in the context of the evolution of cooperation (25–28), but general multiplayer interactions have received less attention (see, however, refs. 9–13).

We again assume there to be two strategies, A and B. We can also maintain the same definition of the payoffs as α_i. As there are d – 1 other individuals, excluding the focal player, i can range from 0 to d – 1. We can depict the payoffs α_i in the form

However, for multiplayer games an additional complication arises. Consider a three-player game (d = 3). Let the focal player be playing A. As d = 3 there are d – 1 = 2 other players. If one of them is of type A and the other of type B, there can be the combinations AAB or ABA. Do these two structures give the same payoffs? Or, in a more general sense, does the order of players matter? If order does matter, the payoffs are in a d-dimensional discrete space, as illustrated by Fig. 1. There are numerous examples where the order of the players is very important. In a game of soccer, it is necessary to have a player specialized as the goal keeper in the team. But it is also important that the goal keeper is at the goal and not acting as a center-forward. A biological example has been studied by Stander in the Etosha National Park (29). The lionesses hunt in packs and employ the flush-and-ambush technique. Some lie in ambush while others flush out the prey from the flanks and drive them toward the ones waiting in ambush. This technique needs more than two players to be successful. Some lionesses always display a particular position to be a preferred one (right flank, left flank, or ambush). The success rate is higher if the lionesses are in their preferred positions. Thus, the ordering of players matters here.

Fig. 1.

For 2 × 2 games, the payoff matrix has 4 entries. If we increase the number of strategies, the payoff matrix grows in size. For example, the payoff matrix of a 3 × 3 game has 9 entries. If we increase the number of players, the payoff matrix becomes higher-dimensional. For example, two-strategy games with three players are described by 2 × 2 × 2 payoff structures with 8 entries. In general, a d-player game with n strategies is described by n^d payoff values.

To address situations in which the order of players matters, we have to make use of a tensor notation for writing down the payoffs which offers the flexibility to include higher dimensions of the payoff matrix. Consider a tensor β with d indices defined as follows: , where the first index denotes the focal player's strategy. Each of the indices represents the strategy of the player in the position denoted by its subscript. The index i can represent any of the n strategies. Thus, the total number of entries will be n^d. This structure is the multiplayer equivalent of a payoff matrix (see ref. 9 and Fig. 1). Consider, for example, a game with three players and two strategies (A and B). If the order of players matters, then the payoff values for strategy A are represented by β_AAA, β_AAB, β_ABA, and β_ABB. This increase in complexity is handled by the tensor notation but is not reflected in the tabular notation (2). But as long as interaction groups are formed at random, we can transform the payoffs such that they can be written in the form of 2 (SI Text). In this case, the payoffs are weighted by their occurrence to calculate the average payoffs. For example, in our three-player games, a₁ has to be counted twice (corresponding to β_AAB and β_ABA). If we would consider evolutionary games in structured populations instead of random-interaction group formation, then the argument breaks down and the tensor notation cannot be reduced.

In the case of d-player games with two strategies, we can then write the average payoff π_A obtained by strategy A in an infinite population as , where x is the fraction of A players. An equivalent equation holds for the average payoff π_B of strategy B. The replicator equation of a two-player game is given by ref. 30:

Obviously, there are two trivial fixed points when the whole population consists of A (x = 1) or B (x = 0). In d-player games, both π_A and π_B can be polynomials of maximum degree d – 1 (see SI Text). This implies that the replicator equation can have up to d – 1 interior fixed points. In the two-strategy case, these points can be either stable or unstable. The maximum number of stable interior fixed points possible is d/2 for even d and (d – 1)/2 for odd d; see also refs. 9 and 10, where it is shown that all these scenarios are also attainable. For d = 2, π_A and π_B are polynomials of degree 1; hence, there can be at most one internal equilibrium, which is either unstable (coordination games) or stable (coexistence games). For d = 3, there can also be a second interior fixed point. If one of them is stable, the other one must be unstable. This can lead to a situation in which A is advantageous when rare (the trivial fixed point x = 0 is unstable), and becomes disadvantageous at intermediate frequencies but advantageous again for high frequencies, as in multiplayer stag hunts (11).

For a d-player game to have d – 1 interior fixed points, the quantities a_k – b_k and a_k+1 – b_k+1 must have different signs for all k. However, this condition is necessary (because the direction of selection can only change d – 1 times if the payoff difference a_k – b_k changes sign d – 1 times), but not sufficient (SI Text). Pacheco and coauthors have studied public goods games in which a threshold frequency of cooperators is necessary for producing any public good (11, 12). The payoff difference changes sign twice at this threshold value and hence there can be at most two internal equilibria.

A d-player game has a single internal equilibrium if a_k – b_k has a different sign from a_k+1 – b_k+1 for a single value of k: In this case, A individuals are disadvantageous at low frequency and advantageous at high frequency (or vice versa). If a_k – b_k changes sign only once, then the direction of selection can change at most once. Thus, this condition is sufficient in infinite populations.

Now we deviate from the replicator dynamics, where the average payoff of a strategy is equated to reproductive fitness, and turn our attention to finite populations. In this case, the sampling for π_A and π_B is no longer binomial but hypergeometric (SI Text). In finite populations, the intensity of selection measures how important the payoff from the game is for the reproductive fitness. We take fitness as an exponential function of the payoff, f_A = exp(+ wπ_A) for A players and f_B = exp(+ wπ_B) for B players (31). If , selection is strong and the average payoffs dictate the outcome of the game, whereas if , then selection is weak and the payoffs have only marginal effect on the game. This choice of fitness recovers the results of the usual Moran process introduced by Nowak et al. (15) and simplifies the analytical calculations significantly under strong selection (31). However, for nonweak selection, other payoffs to fitness mappings lead to slightly different results (18). We employ the Moran process to model the game, but our results hold for any birth-death process in which the ratio of transition probabilities can be approximated under weak selection by a term linear in the payoff difference in addition to the neutral result. In the Moran process, an individual is selected for reproduction at random but proportional to its fitness. The individual produces identical offspring. Another individual is chosen at random for death. With this approach, we can address the basic properties of d-player games with two strategies generalizing quantities from 2 × 2 games.

Does A replace B with a higher probability than vice versa? Comparing the fixation probabilities of a single A or B individual, ρ_A and ρ_B, we find that ρ_A > ρ_B is equivalent to

(SI Text). For d = 2, we recover the risk dominance from above. For large N, the condition reduces to (13)

These two conditions are valid for any intensity of selection in our variant of the Moran process.

The one-third law for two-player games is not valid for a higher number of players (SI Text). Instead, the condition we obtain for the payoff entries is not directly related to the internal equilibrium points (as opposed to the two-player case, which makes the one-third law special). For weak selection, we show in SI Text that ρ_A > 1/N is equivalent to

For large population size this reduces to (13)

which is the one-third law from above for d = 2. Inequality 7 means that the initial phase of invasion is of most importance: The factor d – k decreases linearly with k, and the payoff values with small indices k are more important than the payoff values with larger indices. Thus, the payoffs relevant for small mutant frequencies determine whether the condition is fulfilled. In other words, the initial invasion is crucial to obtain a fixation probability larger than 1/N.

In general, conditions 5 and 7 are independent of each other. When 5 is satisfied and 7 is not satisfied, both fixation probabilities are less than neutral (1/N). But when 5 is not satisfied and 7 is satisfied, both ρ_A and ρ_B are larger than neutral (1/N). This scenario is impossible for two-player games.

Let us now turn to multiplayer games with multiple strategies. As illustrated in Fig. 1, the payoff matrix of a two-player game increases in size when more strategies are added. If more players are added, the dimensionality increases. Now we address the evolutionary dynamics of such games. Again we assume that interaction groups are formed at random, such that only the number of players with a certain strategy—but not their arrangement—matters. The replicator dynamics of a d-player game with n possible strategies can be written as a system of n – 1 differential equations:

where x_j is the frequency of strategy j, π_j is the fitness of strategy j, and is the average fitness. The evolution of this system can be studied on a simplex with n vertices, S_n. The simplex S_n is defined by the set of all of the frequencies which follow the normalization . The fixed points of this system are given by the combination of frequencies of the strategies which satisfy π₁ = ··· = π_n. The vertices of the simplex where x_j is either equal to 1 or 0 are trivial fixed points. In addition, there can be, for example, fixed points on the edges or the faces of the simplex. We speak of fixed points in the interior of the simplex when all payoffs are identical at a point where all frequencies are nonzero, x_j > 0 for all j. The internal equilibria are of special interest, because they may represent points of stable biodiversity. For example, three strains of Escherichia coli competing for resources have been studied (32, 33). K is a killer strain which produces a toxin harmful to S; R does not produce toxin but is resistant to the toxin of K. The sensitive strain S is affected by the toxin of K. These three strains are engaged in a kind of rock-paper-scissors game. K kills S. S reproduces faster than R, not paying the cost for resistance. R is superior to K, being immune to its toxin. The precise nature of interactions determines whether biodiversity is maintained in an unstructured population (30, 34). In our context, this is reflected by the existence of an isolated internal fixed point.

Here we ask the more general question of whether there are internal equilibria in d-player games with n strategies. If so, then how many internal equilibria are possible? It has been shown that for a two-player game with any number of strategies n there can be at most one isolated internal equilibrium (30, 35). In SI Text, we demonstrate that the maximum number of internal equilibria in d players with n strategies is

The maximum possible number of internal equilibria increases as a polynomial in the number of players, but exponentially in the number of strategies. For example, for d = 4 and n = 3, the maximum number of internal equilibria is 9 (see Fig. 2). Note that for d = 2 we recover the well-known unique equilibrium. For n = 2, we recover the maximum of d – 1 internal equilibria (see above). Of course, not all of these equilibria are stable. Broom et al. have shown which patterns of stability are attainable for general three-player three-strategy games (9).

Fig. 2.

Evolutionary dynamics in a simplex with the maximum number of internal equilibria for d = 4 players and n = 3 strategies as given by (d – 1)ⁿ⁻¹ = 9. On the dashed cubic curve, we have π₁ = π₃. On the full cubic curve, we have π₂ = π₃. When both lines intersect in the interior of the simplex, we have an internal equilibrium.

This illustrates that many different states of biodiversity are possible in multiplayer games, whereas in two-player games only a single one is possible. This is a crucial point when one attempts to address the question of biodiversity with evolutionary game theory. In the previous example, the studies dealing with E. coli, consider the system as a d = 2 player game with three strategies. Do we really know that d = 2? If strains are to be engineered to stably coexist, then multiple interactions (d > 2) would open up the possibility of multiple internal fixed points instead of the single one for d = 2.

If we choose a game at random, what is the probability that the game has a certain number of internal equilibria? To this end, we take the following approach: We generate many random payoff structures in which all payoff entries are uniformly distributed random numbers (36). For each payoff structure, we compute the number of internal equilibria. It turns out that games with many internal equilibria are the exception rather than the rule. For example, the probability of seeing two or more internal equilibria in a game with four players and three strategies is ≈24%. The probability that a randomly chosen game has the maximum possible number of equilibria decreases with increasing number of players and number of strategies (see Fig. 3). Also, the probability of having a single equilibrium decreases. Instead, we obtain several internal equilibria in the case of more than two players. For two-player games, the probability of seeing an internal equilibrium at all decreases roughly exponentially with the number of strategies. This poses an additional difficulty in coordinating in multiplayer games, because several different solutions may be possible that look quite similar at first sight.

Fig. 3.

The probabilities of observing the different numbers of internal equilibria, 0 to (d – 1)ⁿ⁻¹, as the system gets more complex in the number of strategies n and the number of players d. Random games are generated by choosing the payoff entries a_k, b_k, … from a uniform distribution. If we consider that the order does matter and generate the random games based on the entries of a payoff structure with n^d entries, then the probability of observing a particular number of equilibria is only slightly lower (averages over 10⁸ different games with uniformly chosen payoff entries a_k, b_k, …).

Discussion

Multiplayer games with multiple strategies is what we find all around. We interact with innumerable people at the same time, directly or indirectly. Some interactions may be pairwise, but others are not. In real life, we may typically be engaged in many-person games instead of a disjoined collection of two-person games (8). The evolution and maintenance of cooperation, problems pertaining from group hunting to deteriorating climate, all are fields for a multiple number of players (29, 37, 28, 38). They can have different interests and hence use different strategies. There are other cases such as the maintenance of biodiversity where multiplayer interactions may lead to a much richer spectrum for biodiversity than the commonly analyzed two-player interactions. The presence of multiple stable states also contributes to the intricate dynamics observed in the maintenance of biodiversity (39). Multiplayer games may help to improve our understanding of such systems. The problem of handling multiple equilibria is not just limited to biological games but also appears in economics (40, 41). Many insights can be obtained by studying two-player games, but it blurs the complexity of multiplayer interactions. Here we have derived some basic rules which apply to multiplayer games with two strategies for finite as well as infinite populations and discussed the number of internal equilibria in d-player games with n strategies which determine how the dynamics proceeds.

This theory can be applied to all kinds of games with any number of players and strategies and can thus be easily applied to public goods games, multiplayer stag hunts, or multiplayer snowdrift games. We believe that this opens up avenues where we can get analytical descriptions of situations which are thought to be very complex, and further discussions of these issues will prove to be fruitful due to the intrinsic importance of multiplayer interactions. We conclude this approach by quoting Hamilton again: “A healthy society should feel sea-sick when confronted with the endless internal instabilities of the ‘solutions’, ‘coalition sets’, etc., which the theory of many-person games has had to describe” (8).