Learning enables adaptation in cooperation for multi-player stochastic games

Abstract

Interactions among individuals in natural populations often occur in a dynamically changing environment. Understanding the role of environmental variation in population dynamics has long been a central topic in theoretical ecology and population biology. However, the key question of how individuals, in the middle of challenging social dilemmas (e.g. the ‘tragedy of the commons’), modulate their behaviours to adapt to the fluctuation of the environment has not yet been addressed satisfactorily. Using evolutionary game theory, we develop a framework of stochastic games that incorporates the adaptive mechanism of reinforcement learning to investigate whether cooperative behaviours can evolve in the ever-changing group interaction environment. When the action choices of players are just slightly influenced by past reinforcements, we construct an analytical condition to determine whether cooperation can be favoured over defection. Intuitively, this condition reveals why and how the environment can mediate cooperative dilemmas. Under our model architecture, we also compare this learning mechanism with two non-learning decision rules, and we find that learning significantly improves the propensity for cooperation in weak social dilemmas, and, in sharp contrast, hinders cooperation in strong social dilemmas. Our results suggest that in complex social–ecological dilemmas, learning enables the adaptation of individuals to varying environments.

1. Introduction

Throughout the natural world, cooperating through enduring a cost to endow unrelated others with a benefit is evident at almost all levels of biological organisms, from bacteria to primates [1]. This phenomenon is especially true for modern human societies with various institutions and nation-states, in which cooperation is normally regarded as the first choice to cope with some major global challenges, such as curbing global warming [2,3] and governing the commons [4]. However, the mechanism underlying cooperative behaviour has perplexed evolutionary biologists and social economists for a long time [5,6]. Since according to the evolutionary theory of ‘survival of the fittest’ and the hypothesis of Homo economicus, this costly prosocial behaviour will be definitively selected against and should have evolved to be dominated by selfish act [7].

To explain how cooperation can evolve and be maintained in human societies or other animal groups, a large body of theoretical and experimental models have been put forward based on evolutionary game theory [6,8,9] and social evolution theory [10]. Traditionally, the vast majority of the previous work addressing this cooperative conundrum concentrates on the intriguing paradigm of a two-player game with two strategies, Prisoner’s Dilemma [6,11]. Motivated by abundant biological and social scenarios where interactions frequently occur in a group of individuals, its multi-person version, the public goods game, has attracted much attention in recent years [12]. Meanwhile, it also prompts a growing number of researchers to devote to studying multi-player games and multi-strategy games [13–18]. However, these prominent studies implicitly assume, as most of the canonical work does, that the game environment is static and independent of players’ actions. In other words, in these models, how players act by choosing game-play strategies only affects the strategic composition in the population, but the game environment itself is not influenced. As a result, a single fixed game is played repeatedly. Of course, this assumption is well grounded, if the timescale of interest (e.g. the time to fixation or extinction of a species) is significantly shorter than that of the environmental change. For most realistic social and ecological systems, however, it seems to be too idealized. Hence, an explicit consideration of environmental change is needed. A prototypical instance is the overgrazing of common pasture lands [19], where the depleted state may force individuals to cooperate and accordingly the common-pool resources will increase, whereas the replete state may induce defection and the common-pool resources will decrease [20,21]. Other examples also exist widely across scales from small-scale microbes to large-scale human societies [22]. A common feature of these examples is the existence of the feedback loop where individual behaviours alter environmental states, and are influenced in turn by the modified environment [20,23].

Although the effect of environmental variations on population dynamics has long been recognized in theoretical ecology and population biology [24–26], it is only recently that there has been a surge of interest in constructing game–environment feedbacks [20,21,23,27,28] to understand the puzzle of cooperation, especially in structured populations [29–31]. Different from the conventional set-up in evolutionary game theory [8,9], the key conceptual innovation of these works is the introduction of multiple games [32,33], evolving games [34,35], dynamical system games [36] or stochastic games [37,38]. By doing so, the players’ payoff depends on not only strategic interactions but also the environmental state, and meanwhile, the fluctuation of the environment will be subject to the actions adopted by players. In this sense, the consideration of a dynamic game environment for the evolution of cooperation has as least two significant implications. First, it vastly expands the existing research scope of evolutionary game theory by adding a third dimension (multiple games) to the prior two-dimension space (multiple players and multiple strategies) [33]. That is, this extension generalizes the existing framework to encompass a broader range of scenarios. Second, the new key component, environmental feedbacks [20,23], is integrated seamlessly into the previous theoretical architecture.

While these promising studies primarily focused on pre-specified or pre-programmed behavioural policies to analyse the interdependent dynamics between individual behaviours and environmental variations, the key question of how individuals adjust their behaviours to adapt to the changing environment has not yet been sufficiently addressed. In fact, when confronting complex biotic and abiotic environmental fluctuations, how organisms adaptively modulate their behaviours is of great importance for their long-term survival efforts [25,39]. For example, those plants growing in the lower strata of established canopies can adjust their stem elongation and morphology in response to the spectral distribution of radiation, especially the ratio of red to far-red wavelength bands [40]; in arid regions, bee larvae, as well as angiosperm seeds, strictly comply with a bet-hedging emergence and germination rule such that reproduction activities are only limited to a short period of time following the desert rainy season [41]. Particularly, as an individual-level adaptation, learning through reinforcement is a fundamental cognitive or psychological mechanism used by humans and animals to guide action selections in response to the contingencies provided by the environment [42–44]. Employing the experience gained from historical interactions, individuals always tend to reinforce those actions that will increase the probability of rewarding events and lower the probability of aversive events. Although this learning principle has become a central method in various disciplines, such as artificial intelligence [44,45], neuroscience [43], learning in games [46] and behavioural game theory [47], there is still a lack of theoretical understanding of how it guides individuals to make decisions in order to resolve cooperative dilemmas.

In the present work, we develop a general framework to investigate whether cooperative behaviours can evolve by learning through reinforcement in constantly changing multi-player game environments. To characterize the interplay between players’ behaviours and environmental variations, we propose a normative model of multi-player stochastic games, in which the outcome of one’s choice relies on not only the opponents’ choices but also the current game environment. Moreover, we use a social network to capture the spatial interactions of individuals. Instead of using a pre-specified pattern, every decision-maker in our model learns to choose a behavioural policy by associating each game outcome with reinforcements. By doing so, our model not only considers the environmental feedback, but also incorporates a cognitive or psychological feedback loop (i.e. players’ decisions determine their payoffs in the game, and in turn are affected by the payoffs). When selection intensity is so weak that the action choices of players are just slightly influenced by past reinforcements, we derive the analytical condition that allows for cooperation to evolve under the threat of the temptation to defection. Through extensive agent-based simulations, we validate the effectiveness of the closed-form criterion in well-mixed and structured populations. Also, we compare the learning mechanism with two non-learning decision rules, and interestingly, we find that learning markedly improves the propensity for cooperation in weak social dilemmas whereas it hinders cooperation in strong social dilemmas. Furthermore, under non-stationary conditions, we analyse how cooperation coevolves with the environment and the effect of external incentives on the cooperative evolution by agent-based simulations.

2. Model and methods

2.1. Model

We consider a finite population of N individuals living in an evolving physical or social environment. The population structure describing how individuals interact with their neighbours is characterized by a network, where nodes represent individuals and edges indicate interactions. When individuals interact with their neighbours, only two actions, cooperation (C) and defection (D), are available, and initially, every individual is initialized with a random action in the set $\mathcal{A}=\{C,D\}$ with a certain probability. In each time step, one individual is chosen randomly from the population to be the focal player, and then d − 1 of its neighbours as co-players are selected at random to form a d-player (d ≥ 2) stochastic game [37,38]. To ensure that the game can always be organized successfully, we assume that each individual in the population has at least d − 1 neighbours. Denote the possible number of C players among d − 1 co-players by the set $\mathcal{J}\triangleq \{0,1,\dots ,d-1\}$, and possible environmental states by the set $\mathcal{S}\triangleq \{s^1,s^2,\dots ,s^M\}$, where sⁱ, i = 1, 2, …, M, represents the environmental state of type i. Then, depending on the co-players’ configuration $j\in \mathcal{J}$ and the environmental state $s\in \mathcal{S}$ in the current round, each player will gain a payoff given in table 1. Players who take action C will get a payoff ${\text{a}}_{\hspace{0.17em}j}(s)\in \mathbb{R}$, whereas those who take action D will get a payoff $b_{\hspace{0.17em}j}(s)\in \mathbb{R}$, where $\mathbb{R}$ represents the set of real numbers. Players update their actions asynchronously; that is, in each time step, only the focal player updates its action, and other individuals still use the actions in the previous round. Furthermore, to prescribe the action update rule, we define the policy $\pi (s,j,a;\theta ,\beta )\hspace{0.17em}:\hspace{0.17em}\mathcal{S}\times \mathcal{J}\times \mathcal{A}\to [0,1]$ with two parameters θ and β to specify the probability that action a is chosen by the focal player when there are j opponents taking action C among d − 1 co-players in the environmental state $s\in \mathcal{S}$. Therein, $\theta \in {\mathbb{R}}^L$ is the column vector of L-dimension used for updating the policy by learning through reinforcement, and β ∈ [0, + ∞) is the selection intensity [48], also termed the adaptation rate [49], which captures the effect of past reinforcements on the current action choice.

Table 1.

Payoff table of the d-player stochastic game.

no. C co-players	d − 1	…	j	…	0
C	ad−1(s)	…	aj(s)	…	a0(s)
D	bd−1(s)	…	bj(s)	…	b0(s)

After each round, players’ decisions regarding whether to cooperate or defect in the game interaction will not only influence their immediate payoffs but also the environmental state in the next round. That is to say, the probability of the environmental state in the next round is conditioned on the action chosen by the focal player and the environmental state in the current round. Without loss of generality, we here assume that the dynamics of environmental states {s_t} obey an irreducible and aperiodic Markov chain, which thus possesses a unique stationary distribution. Also, from table 1, it is clear that the payoff of each player is a function of the environmental state. Therefore, when the environment transits from one state to another, the type of the (multi-player) normal-form game defined by the payoff table may be altered accordingly.

The emergence of the new environmental state in the next round, apart from influencing the game type, may also trigger players to adjust their behavioural policies. This is because those previously used decision-making schemes may no longer be appropriate in the changed environment. We here consider a canonical learning mechanism, the actor–critic reinforcement learning [42–44], to characterize the individual adaptation to the fluctuating environment. Specifically, after each round, the players’ payoffs received from the game interaction will play a role of the incentive signal of the interactive scenario. If one choice gives rise to a higher return in a certain scenario, then it will be reinforced with a higher probability in the future when encountering the same situation again. By contrast, those choices resulting in lower payoffs will be weakened gradually. Technically, this process is achieved via updating the learning parameter θ of the policy after each round (see Methods for more details). In the successive round, the acquired experience will be shared within the population and the updated policy will be reused by the newly chosen focal player to determine which action to be taken. In a similar way, this dynamical process of game formation and policy updating is repeated infinitely (figure 1).

Figure 1.

Illustration of evolutionary dynamics for 4-player stochastic games in the structured population. (a) At a time step t, a random individual is chosen as the focal player (depicted by the dashed red circle), and then three of its neighbours are selected randomly as co-players to form a 4-player game (because the focal player only has three neighbours, all of them are chosen), which is depicted by the light magenta shaded area. Conditioned on the focal player’s action and the environmental state s_t at time t, the environmental state at time t + 1 will be changed to s_t+1 with a transition probability. Similarly, a new round of the game will be reorganized at time t + 1. This process is repeated infinitely. (b) At time t, after perceiving the environmental state s_t and the co-players’ configuration j, the focal player uses policy π to determine which action to be taken, whereas its co-players still use their previous actions in the past round. At the end of this round, each player will gain a payoff, which will play a role of the feedback signal and will assist the focal player to update its policy.

2.2. Methods

2.2.1. Actor–critic reinforcement learning

As the name suggests, the architecture of the actor–critic reinforcement learning consists of two modules. The actor module maintains and learns the action policy. Generally, there are two commonly used forms, ε-greedy and Boltzmann exploration [44,45]. Here, we adopt the latter for convenience, and consider the following Boltzmann distribution with a linear combination of features:

$$\pi (s,j,a;\theta ,\beta )=\frac{{\text{e}}^{\beta {\theta }^{\mathrm{T}}{\varphi }_{s,j,a}}}{\sum _{b\in \mathcal{A}}\hspace{0.17em}{\text{e}}^{\beta {\theta }^{\mathrm{T}}{\varphi }_{s,j,b}}},\hspace{0.17em}\mathrm{\forall }\hspace{0.17em}s\in \mathcal{S},j\in \mathcal{J},a\in \mathcal{A},$$ 2.1

where ${\varphi }_{s,j,a}\in {\mathbb{R}}^L$ is the column feature vector with the same dimension of θ, which is handcrafted to capture the important features when a focal player takes action a given the environmental state s and the number of C players j among its d − 1 co-players. Moreover, the dimension of the feature vector will in general be chosen to be much smaller than that of environmental states for the computational efficiency, i.e. L ≪ M. For the construction of the feature vector, there are many options, such as polynomials, Fourier basis, radial basis functions and artificial neural networks [44]. As mentioned in the Model, β controls the selection intensity, or equivalently the adaptation rate. If β → 0, it defines a weak selection and the action choice is only slightly affected by past reinforcements. When β = 0, in particular, players choose actions with uniform probability. By contrast, if β → +∞, the action with the maximum θ^Tϕ_s,j,a will be exclusively selected.

Another module is the critic, which is designed to evaluate the performance of the policy. In general, the long-run expected return of the policy per step, ρ(π), will be a good measurement of the policy’s performance, which is defined by

$$\rho (\pi )\triangleq \underset{t\to \mathrm{\infty }}{lim}\frac{1}{t}\mathbb{E}\{r_1+r_2+\cdots +r_t|\pi \},$$ 2.2

where r_t+1 ∈ {a_d−1(s), … , a₀(s), b_d−1(s), … , b₀(s)} is a random variable which denotes the payoff of the focal player at time t ∈ {0, 1, 2, … }. In particular, if one denotes the probability that the environmental state at time t is s_t under the policy π when starting from the initial state s₀ by Pr{s_t = s|s₀, π}, and the average probability that all possible individuals chosen as the focal player encounter j opponents taking action C among d − 1 co-players by p_·j, then ρ(π) can be computed by

$$\rho (\pi )=\sum _{s\in \mathcal{S}}{d}^{\pi }(s)\sum _{\hspace{0.17em}j\in \mathcal{J}}{p}_{\cdot j}\sum _{a\in \mathcal{A}}\pi (s,j,a;\theta ,\beta ){\mathcal{R}}_{s,j}^a,$$ 2.3

where $d^{\pi }(s)=\underset{t\to \mathrm{\infty }}{lim}Pr\{s_t=s|s_0,\pi \}$ is the stationary distribution of environmental states under the policy π; ${\mathcal{R}}_{s,j}^a$ is the payoff of the focal player when it takes action a given the environmental state s and the number of C players j among its d − 1 co-players, which is given by

$${\mathcal{R}}_{s,j}^a=\left\{\begin{array}{cc}{a}_j(s),& \hspace{0.17em}\text{if }\mathit{a}=C;\\ b_j(s),& \hspace{0.17em}\text{if }a=D.\end{array}\right.$$ 2.4

Moreover, to measure the long-term accumulative performance of the policy, we define a Q-value function,

$$\begin{array}{rl}{Q}^{\pi }(s,j,a)& \triangleq \sum _{t=1}^{\mathrm{\infty }}\mathbb{E}\{r_t-\rho (\pi )|s_0=s,j_0=j,a_0=a,\pi \},\hspace{0.17em}\mathrm{\forall }\hspace{0.17em}s\in \mathcal{S},\\ & j\in \mathcal{J},a\in \mathcal{A},\end{array}$$ 2.5

which is a conditional value depending on the initial environmental state s₀ = s, the number of C players j₀ = j among d − 1 co-players, and action a₀ = a at time t = 0. Since the space of the environmental state is usually combinatorial and extremely large in many game scenarios, it is in effect impossible to calculate the Q-value function exactly for every environmental state within finite time with given computational resources [44]. Typically, one effective way to deal with this problem is to find a good approximation of the Q-value function. Let f_w(s, j, a) be the approximation to the Q-value function and satisfy the compatibility condition [50,51]

$$\begin{array}{rl}\hspace{0.17em}{f}_w(s,j,a)& \hspace{0.17em}=w^{\mathrm{T}}\left[\frac{\mathrm{\partial }\pi (s,j,a;\theta ,\beta )}{\mathrm{\partial }\theta }\frac{1}{\pi (s,j,a;\theta ,\beta )}\right]\\ & \hspace{0.17em}=w^{\mathrm{T}}\left[{\varphi }_{s,j,a}-\sum _{b\in \mathcal{A}}\pi (s,j,b;\theta ,\beta ){\varphi }_{s,j,b}\right]\beta ,\end{array}$$ 2.6

where $w\in {\mathbb{R}}^L$ is the column vector of weight parameters. To effectively approximate the Q-value function, it is natural to learn f_w(s, j, a) by updating w via the least mean square method under the policy π. After acquiring the approximated measurement of the policy’s performance f_w(s, j, a), policy π can be then improved by following the gradient ascent of ρ(π). Thus, the full algorithm of the actor–critic reinforcement learning can be given by (see electronic supplementary material SI.1 for details)

$$\left.\begin{array}{rl}& w_{t+1}=w_t+{\alpha }_t[r_{t+1}-{\overline{R}}_t+f_{w_t}(s_{t+1},j_{t+1},a_{t+1})-f_{w_t}(s_t,j_t,a_t)]\frac{\mathrm{\partial }{f}_{w_t}(s_t,j_t,a_t)}{\mathrm{\partial }{w}_t},\\ & {\theta }_{t+1}={\theta }_t+{\gamma }_t\frac{\mathrm{\partial }\pi (s_t,j_t,a_t;{\theta }_t,\beta )}{\mathrm{\partial }{\theta }_t}\frac{1}{\pi (s_t,j_t,a_t;{\theta }_t,\beta )}{f}_{w_t}(s_t,j_t,a_t),\end{array}\right\}$$ 2.7

where ${\overline{R}}_t$ is the estimation of ρ(π), and iterates through ${\overline{R}}_{t+1}={\overline{R}}_t+[r_{t+1}-{\overline{R}}_t]/(t+1)$ and ${\overline{R}}_0=0$, t = 0, 1, 2, … ; α_t and γ_t are learning step-sizes which are positive, non-increasing for $\mathrm{\forall }\hspace{0.17em}t$, and satisfy $\sum _t{\alpha }_t=\sum _t{\gamma }_t=\mathrm{\infty }$, $\sum _t{\alpha }_t^2<\mathrm{\infty }$ and $\sum _t{\gamma }_t^2<\mathrm{\infty }$, and ${\gamma }_t/{\alpha }_t\to 0\text{ for }t\to \mathrm{\infty }$. These conditions required for the learning step-sizes guarantee that the policy parameter θ_t is updated at a slower timescale than that of the function approximation w_t, and thus ensure the convergence of the learning rule [51,52].

2.2.2. Evolution of cooperative behaviours

To capture the evolutionary process of cooperation, we first denote the number of C players in the population by n_t at time t. Since there is only one individual to revise its action per step in our model, all possible changes of n_t in each time step will be limited to increasing by one, decreasing by one, or keeping unchanged. It implies that the evolutionary process of cooperation can be formulated as a Markov chain {n_t} defined over the finite state space $\mathcal{N}=\{0,1,2,\dots ,N\}$. Meanwhile, the transition probability from $n_t=u\in \mathcal{N}$ to $n_{t+1}=v\in \mathcal{N}$ can be calculated by

$$\begin{array}{rl}& p_{u,v}(t)=\sum _{s\in \mathcal{S}}Pr\{s_t=s|s_0,\pi \}\hspace{0.17em}\sum _{\hspace{0.17em}j\in \mathcal{J}}\left\{\begin{array}{cc}{p}_C{p}_{C,j}\pi (s,j,C;{\theta }_t,\beta )+p_D{p}_{D,j}\pi (s,j,D;{\theta }_t,\beta ),& \text{ for }v=u;\\ p_C{p}_{C,j}\pi (s,j,D;{\theta }_t,\beta ),& \text{for }v=u-1;\\ p_D{p}_{D,j}\pi (s,j,C;{\theta }_t,\beta ),& \text{for }v=u+1;\\ 0,& \text{otherwise},\end{array}\right.\end{array}$$ 2.8

where p_C = u/N (respectively, p_D = (N − u)/N) is the probability that an individual who previously took action C (respectively, D) is chosen as the focal player at time t; p_C,j (respectively, p_D,j) is the average probability that players who previously took action C (respectively, D) encounter j opponents taking action C among d − 1 co-players at time t. It is clear that the Markov chain is non-stationary because the transition probabilities change with time.

To find the average abundance of cooperators in the population, we first note that the actor–critic reinforcement learning converges [50,51] and the environmental dynamics have been described by an irreducible and aperiodic Markov chain. As such, we denote the limiting value of the policy parameter θ_t for t → ∞ by θ* (a local optimum of ρ(π); see electronic supplementary material SI.1 for details), and the unique stationary distribution of environmental states by $d^{\pi }(s)=\underset{t\to \mathrm{\infty }}{lim}Pr\{s_t=s|s_0,\pi \}$. It follows that the probability transition matrix P(t) = [p_u,v(t)]_(N+1)×(N+1) will converge to $P^{\ast }={[p_{u,v}^{\ast }]}_{(N+1)\times (N+1)}$ for t → ∞, where

$$\begin{array}{rl}\hspace{0.17em}{p}_{u,v}^{\ast }& \hspace{0.17em}=\underset{t\to \mathrm{\infty }}{lim}{p}_{u,v}(t)=\sum _{s\in \mathcal{S}}{d}^{\pi }(s)\sum _{\hspace{0.17em}j\in \mathcal{J}}\left\{\begin{array}{cc}{p}_C{p}_{C,j}\pi (s,j,C;{\theta }^{\ast },\beta )+p_D{p}_{D,j}\pi (s,j,D;{\theta }^{\ast },\beta ),& \text{for }v=u;\\ p_C{p}_{C,j}\pi (s,j,D;{\theta }^{\ast },\beta ),& \text{for }v=u-1;\\ p_D{p}_{D,j}\pi (s,j,C;{\theta }^{\ast },\beta ),& \text{for }v=u+1;\\ 0,& \text{otherwise}.\end{array}\right.\end{array}$$ 2.9

Moreover, it is noteworthy that the Markov chain described by the probability transition matrix P* will be irreducible and aperiodic. This is because based on the matrix P*, any two states of the Markov chain are accessible to each other and the period of all states is 1. Hence, one can conclude that the non-stationary Markov chain {n_t} is strongly ergodic [53,54], and there exists a unique long-run (i.e. stationary) distribution $X={[x_n]}_{1\times (N+1)},n\in \mathcal{N}$. Therein, X can be obtained by calculating the left eigenvector corresponding to eigenvalue 1 of the probability transition matrix P*, i.e. the unique solution to X(P* − I) = 0_N+1 and $\sum _{n\in \mathcal{N}}{x}_n=1$, where I is the identity matrix with the same dimension of P* and 0_N+1 is the row vector with N + 1 zero entries. When the system has reached the stationary state, the average abundance of C players in the population can be computed by $⟨x_C⟩=\sum _{n\in \mathcal{N}}(x_n\cdot n/N)$. If 〈x_C〉 > 1/2, it implies that C players are more abundant than D players in the population.

3. Results

3.1. Conditions for the prevalence of cooperation

We first study the condition under which cooperation can be favoured over defection, and restrict our analysis in the limit of weak selection (β → 0) given that finding a closed-form solution to this problem for arbitrary selection intensity is usually NP-complete or # P-complete [55]. In the absence of mutations, such a condition can be obtained in general by comparing the fixation probability of cooperation with that of defection [48]. In our model, however, how players update their actions is conducted by the policy with an exploration–exploitation trade-off, which possesses a property similar to the mutation–selection process [56]. Thus, in this case, we need to calculate the average abundance of C players when the population has reached the stationary state, and determine whether it is higher than that of D players [57]. Using all a_j(s) to construct the vector A = [a(s¹), a(s²), … , a(s^M)]^T, and all b_j(s) to construct the vector B = [b(s¹), b(s²), … , b(s^M)]^T, where a(s^k) = [a₀(s^k), a₁(s^k), … , a_d−1(s^k)] and b(s^k) = [b_d−1(s^k), b_d−2(s^k), … , b₀(s^k)], k = 1, 2, … , M, it follows that under weak selection the average abundance of C players in the stationary state is (see electronic supplementary material SI.2 for details)

$$⟨x_C⟩=\frac{1}{2}+\frac{1}{N}\left[\sum _{s\in \mathcal{S}}{d}^{\pi }(s){\theta }^{\ast \mathrm{T}}{\mathit{\Phi }}_s(A-B)\right]\beta +o(\beta ),$$ 3.1

and thus it is higher than that of D players if and only if

$$\sum _{s\in \mathcal{S}}{d}^{\pi }(s){\theta }^{\ast \mathrm{T}}{\mathit{\Phi }}_s(A-B)>0,$$ 3.2

where Φ_s is the coefficient matrix corresponding to the environmental state s, and needed to be calculated for the given population structure, but independent of both a_j(s) and b_j(s) for $\mathrm{\forall }\hspace{0.17em}j\in \mathcal{J}$ and $\mathrm{\forall }s\in \mathcal{S}$.

To obtain an explicit formulation of condition (3.2), we further consider two specific population structures, well-mixed populations and structured populations. In the former case, the interactive links of individuals are described by a complete graph, whereas in the latter case, they are described by a regular graph with node degree d − 1. When the population size is sufficiently large, we find that in the limit of weak selection, condition (3.2) in these two populations reduces to an identical closed form (see electronic supplementary material SI.3 for details):

$$\sum _{s\in \mathcal{S}}{d}^{\pi }(s)\sum _{\hspace{0.17em}j=0}^{d-1}\left(\genfrac{}{}{0ex}{}{d-1}{j}\right)\frac{1}{2^{d+1}}{\theta }^{\ast \mathrm{T}}[{\varphi }_{s,j,C}-{\varphi }_{s,j,D}]>0.$$ 3.3

Through extensive agent-based simulations, we validate the effectiveness of this criterion. As illustrated in figure 2,we calculate the average abundance of C players in the population with two distinct environmental states, s¹ and s², which, for instance, can represent the prosperous state and degraded state of a social–ecological system [20,58], respectively. To specify the type of the (multi-player) normal-form game defined by the payoff table 1 for each given environmental state, in figure 2, we consider that one of the three candidates, the public goods game (PGG) [19], threshold public goods game (TPGG) [3,59] and d-player snowdrift game (dSD) [60], is played in each state. In these three kinds of games, the implication of defection is unambiguous and it means not to contribute. However, in defining cooperation and calculating payoffs, there are some differences. In the PGG, action C means contributing a fixed amount c to the common pool. After a round of donation, the sum of all contributions from the d-player group will be multiplied by a synergy factor r_s > 1 and then allotted equally among all members, where r_s depends on the current game environment s. In this case, the payoffs of cooperators and defectors are computed by a_j(s) = (j + 1)r_sc/d − c and b_j(s) = jr_sc/d, $j\in \mathcal{J}$, respectively. The aforementioned setting is also true for the TPGG, except that there exists a minimum contribution effort, T, for players to receive benefits. More specifically, only when the number of C players in the d-player game is not smaller than T can each player receive a payoff from the common pool; otherwise, everyone gets nothing. It then follows that a C player will receive a payoff a_j(s) = (j + 1)cr_s/d − c for j ≥ T − 1 and a_j(s) = 0 otherwise, whereas a D player will receive b_j(s) = jcr_s/d for j ≥ T and b_j(s) = 0 otherwise. Different from the PGG and TPGG, in the dSD, action C means endowing everyone with a fixed payoff ${\mathcal{B}}_s$ and simultaneously sharing a total cost $\mathcal{C}$ evenly with the other C players, where ${\mathcal{B}}_s$ depends on the environmental state s. In this case, the payoffs of cooperators and defectors are then changed to $a_j(s)={\mathcal{B}}_s-\mathcal{C}/(j+1)$ for $j\in \mathcal{J}$, and $b_j(s)={\mathcal{B}}_s$ for j > 0 and b₀(s) = 0, respectively. As shown in figure 2, the theoretical predictions for the average abundance of C players are highly consistent with simulation results, which suggests that criterion (3.3) is effective for determining whether cooperation can outperform defection.

Figure 2.

Average abundance of C players in the population as a function of the stationary proportion of different games. In each homogeneous environmental state, s¹ or s², one of the three normal-form games, PGG, TPGG and dSD, is played. In the top row, three transition graphs are depicted to describe how the environment transits from one state to another. Corresponding to these three transition graphs, the bottom row shows the average abundance of C players in various population structures, based on theoretical calculations and simulations. All simulations are obtained by averaging 40 network realizations and 10⁸ time steps after a transient time of 10⁷, and θ is normalized per step to unify the magnitude. The feature vector ϕ_s,j,a is chosen to be the one-hot vector. Parameter values: N = 400, β = 0.01, $\mathcal{C}=c=1$, r_s = 3 in the PGG while r_s = 4 in the TPGG, ${\mathcal{B}}_s=12$ in (b) while ${\mathcal{B}}_s=4$ in (c), and T = [d/2] + 1 ([ · ] represents the integer part).

Moreover, conditions (3.2) and (3.3) offer us an intuitional theoretical interpretation of why the environment can mediate social dilemmas [22]. As shown in figure 2, in an identical scenario, the average abundance of C players is always less than 1/2 in the homogeneous state where the PGG is played, whereas it is greater than 1/2 in some homogeneous states where a TPGG or dSD is played. The reason is that the social dilemma in the TPGG and dSD is weaker than that in the PGG. Thus, cooperation in these two kinds of games is easier to evolve. Namely, if the environment is homogeneous, condition (3.2) or (3.3) in the PGG is more difficult to be satisfied in contrast to the TPGG or dSD. Due to the existence of the underlying transition of the environment, however, the population may have some opportunities to extricate itself from those hostile environmental states where defection is dominant (e.g. the state of the PGG). This case is especially likely after some prosocial behaviours have been implemented by players [21,29,58]. As such, the population will spend some time staying in the states where defection is not always favourable (e.g. the TPGG or dSD). Consequently, the changing environment balances the conditions that favour versus undermine cooperation, and meanwhile the social dilemma that the population is confronted with is diluted. Such an observation is also in line with the fact that the final condition of whether cooperation can prevail is a convex combination of those results in each homogeneously environmental state, as shown in conditions (3.2) and (3.3).

3.2. Learning versus non-learning

Here, we exclude the effect of reinforcement learning, and apply our model framework to study two prototypical non-learning processes of action choices, the smoothed best response [11] and the aspiration-based update [59,61]. For the former, in each time step, the focal player chosen in our model revises its action by comparing the payoff of cooperation with that of defection, and the more profitable action will be adopted. Instead of doing this in a deterministic fashion, in many real-life situations, it is more reasonable to assume that the choice of the best response is achieved smoothly and influenced by noise. One typical form to model this process is the Fermi function [11]:

$$\begin{array}{r}\pi (s,j,a;\beta )=\frac{1}{1+{\text{e}}^{-\beta [{\mathcal{R}}_{s,j}^a-{\mathcal{R}}_{s,j}^b]}},\hspace{0.17em}\mathrm{\forall }\hspace{0.17em}s\in \mathcal{S},j\in \mathcal{J},a,b(\ne a)\in \mathcal{A},\end{array}$$ 3.4

which specifies the probability for the focal player to choose action $a\in \mathcal{A}$. For the latter, however, the focal player determines whether to switch to a new action by comparing the action’s payoff with an internal aspiration level. If the payoff is higher than the aspiration level, the focal player will switch to that action with a higher probability. Otherwise, its action is more likely to keep unchanged. Similarly, the commonly used form to quantify the probability that the focal player switches to the new action $a\in \mathcal{A}$ is still the Fermi function [59,61]:

$$\pi (s,j,a;\beta )=\frac{1}{1+{\text{e}}^{-\beta [{\mathcal{R}}_{s,j}^a-\mathcal{E})]}},\hspace{0.17em}\mathrm{\forall }\hspace{0.17em}s\in \mathcal{S},j\in \mathcal{J},a\in \mathcal{A},$$ 3.5

where a constant aspiration level $\mathcal{E}$ is adopted because heterogenous aspirations [61] or time-varying aspirations (see electronic supplementary material SI.4) cannot result in altering the evolutionary outcome under weak selection. Using these two non-learning update rules as the decision-making policy of the focal player, under our model framework, we find that in the limit of weak selection, cooperation is more abundant than defection if and only if

$$\sum _{s\in \mathcal{S}}{d}^{\pi }(s)\sum _{\hspace{0.17em}j\in \mathcal{J}}{\sigma }_j[a_j(s)-b_{d-1-j}(s)]>0,$$ 3.6

where σ_j, $\mathrm{\forall }j\in \mathcal{J}$, are some coefficients needed to be calculated for the given population structure, but independent of both a_j(s) and b_j(s). In either well-mixed populations or structured populations, we find that the coefficients are ${\sigma }_j=\left(\genfrac{}{}{0ex}{}{d-1}{j}\right)/2^{d+1}$ for the smoothed best response and ${\sigma }_j=\left(\genfrac{}{}{0ex}{}{d-1}{j}\right)/2^{d+2}$ for the aspiration-based update (see electronic supplementary material SI.4 for details). In particular, if the population consistently stays in a fixed environment, condition (3.6) will reduce to the ‘sigma-rule’ of multi-player normal-form games [15].

In a population where there are three distinct environmental states and in each state one of the PGG, TPGG and dSD is played, we compare the results obtained by learning through reinforcement with those obtained from the two non-learning updates. As illustrated in figure 3, we calculate the average abundance of C players and the expected payoff of focal players per round for all possible stationary distributions of environmental states. Intriguingly, one can find that learning enables players to adapt to the varying environment. When the population stays in the environment where players are confronted with a weak social dilemma (i.e. the TPGG or dSD will be more likely to be played than the PGG), learning players will have a higher propensity for cooperation than those non-learning players. Meanwhile, they will reap a higher expected payoff per step. By contrast, when the population stays in the environment where the social dilemma is strong (i.e. the PGG will be more likely to be played than the TPGG and dSD), learning players will have a lower propensity for cooperation and accordingly they will get a lower expected payoff per step than non-learning players. Once again, we demonstrate that the analytical results are consistent with the agent-based simulations (see electronic supplementary material, figure S5).

Figure 3.

Differences in the average abundance of C players and the expected payoff of players per round between the reinforcement learning (RL) and two non-learning updates. In (a,d), we show the average abundance of C players and the expected payoff per round when players update actions via the RL, respectively. Taking them as the benchmark, (b,e) illustrate the differences between the RL and the smoothed best response (BR), while (c,f) show the gaps between the RL and the aspiration-based rule (aspiration). The population structure is a lattice network (see electronic supplementary material, figures S1–S4, for other population structures with different network degrees). Parameter values: N = 400, d = 5, β = 0.01, $\mathcal{C}=c=1$, T = [d/2] + 1, ${\mathcal{B}}_s=12$, r_s = 3 in the PGG while r_s = 4 in the TPGG.

3.3. Evolutionary dynamics under non-stationary conditions

The aforementioned analysis mainly focuses on the stationary population environment, i.e. the dynamics of environmental states have a unique stationary distribution and the payoff structure of the game does not change in time. Here, we relax this set-up to study the evolutionary dynamics of cooperation under two kinds of non-stationary conditions by agent-based simulations.

3.3.1. Non-stationary environmental state distribution

The first case that we are interested in is that the probability distribution of environmental states changes with time. In a population with two environmental states, s¹ and s², we denote the average proportion of the time that the environment stays in state s¹ (i.e. the average probability that the environment stays in s¹ per step) by z ∈ [0, 1]. Then, the average fraction of time in state s² is 1 − z. To describe the type of game played in each environmental state, let s¹ be the prosperous state where environmental resources are replete and players are at the risk of the ‘tragedy of the commons’ (i.e. a PGG is played), whereas s² be the degraded state where environmental resources are gradually depleted. In both environmental states, cooperation is an altruistic behaviour that will increase the common-pool resources, whereas defection is a selfish behaviour that will lead the common-pool resources to be consumed. Furthermore, the state of common-pool resources (i.e. the environmental state) will conversely affect individual behaviours. To characterize this feedback relation, we here adopt the difference form of the replicator dynamics with environmental feedbacks [20,23] to describe the evolution of the average time proportion of state s¹:

$$\mathrm{\Delta }z(t)=\eta z(t)(1-z(t))(x_C(t)-{\overline{x}}_C),$$ 3.7

where η denotes the positive step-size, x_C(t) is the proportion of C players in the population at time t, and ${\overline{x}}_C$ is the tipping point of the proportion of C players. If the proportion of C players x_C(t) is above the tipping point ${\overline{x}}_C$, it means that the number of cooperators is competent to sustain the supply of common-pool resources. At the same time, the environment will be more likely to stay in the prosperous state s¹, leading z(t) to increase. Otherwise, cooperators are insufficient and the public resources will be continuously consumed. In this case, z(t) will decrease as the environment will more frequently stay in the degraded state s².

We consider that in the prosperous state s¹ players play a PGG. However, in the degraded state s², one of four different games, the PGG, IPGG (inverse public goods game, which reverses the payoffs of action C and D in the PGG), dSH (d-player stag hunt game, which is a variant of the TPGG, and whose only difference from the TPGG is that cooperators always entail a cost c even if j < T), and dSD, is played. The reason that we select these four types of games is twofold. On the one hand, they are commonly used to mimic the essence of a vast number of real-life group interactions [12]; on the other hand, they encompass all possible evolutionary behaviours for the frequency-dependent selection between C and D under the classic replicator dynamics [9]: D dominance, C dominance, bistability, and coexistence (figure 4). Through agent-based simulations, in figure 4, we show the co-evolutionary dynamics of cooperation and the environment under moderate selection intensity. Depending on the game type and the value of the tipping point ${\overline{x}}_C$, the population emerges various dynamic behaviours. Particularly, although our model is stochastic and incorporates the effect of environment and learning, we can still observe those dominance, bistability and coexistence behaviours analogously obtained under the deterministic replicator dynamics. In addition, when replicator dynamics predict that cooperation will be the dominant choice in the degraded state s², our results show some persistent oscillations between cooperation and the environment (panel I in figure 4).

Figure 4.

Co-evolutionary dynamics of cooperation and the environment under moderate selection intensity. From panel I to panel IV, the PGG is fixed to be played in state s¹, while in state s², the IPGG, dSH, PGG and dSD are played, respectively. Under replicator dynamics [12,60,62], the gradients of selection in these four games are shown in each panel, respectively. Blue solid circles are used to depict stable equilibria, while open blue circles are used to depict unstable equilibria. The direction of evolution is indicated by arrows. The phase graphs in each panel show the co-evolutionary dynamics of the time proportion of the PGG and the average proportion of C players for different value intervals of the tipping point ${\overline{x}}_C$. Corresponding to the value interval $0<{\overline{x}}_C<1$, the first row in panel I shows the persistent oscillations of cooperation and the environment. The bottom right sub-figure in panel I shows the linear relation between the average abundance of C players and the average time proportion of the PGG, which suggests that condition (3.3) is still valid for relatively moderate selection intensity. The first row in panel II uses the parameter condition [62] under which there is a stable and an unstable interior equilibrium for the dSH under replicator dynamics (the bottom left), whereas the second row uses that under which there is a unique interior unstable equilibrium (the bottom right). The population structure is a complete graph. Parameter values: N = 400, d = 5, β = 2, $\mathcal{C}=c=1$, ${\mathcal{B}}_s=12$, r_s = 3 for all panels, except in panel II, where r_s = 4 and T = 3 for the first row, and r_s = 5.01 and T = 4 for the second row.

3.3.2. External incentives

Another interesting case is the existence of external incentives, which will undermine the stationarity of the payoff structure of the game. Like two sides of a coin, reward and punishment are two diametrically opposed external incentives for sustaining human cooperation [63,64]. The former is a type of positive incentives where players who cooperate will get an additional bonus, while the latter is a kind of negative incentives where those who defect will be sanctioned and need to pay a fine. At a certain moment during the evolution of cooperation, we separately implement punishment and reward, or jointly enforce them to all players in the population with four environmental states. One can observe that both punishment and reward are effective tools in promoting cooperation, even if the game environment may change (figure 5).

Figure 5.

Evolution of cooperation under the influence of external incentives. Light solid lines indicate simulations whereas dash dot lines are theoretical results. During the evolution, we separately implement punishment (the fine is 0.4) and reward (the bonus is 0.65), or jointly enforce them in a population where the IPGG, dSH, PGG and dSD are played in each state with probability distribution (0.05, 0.05, 0.85, 0.05), respectively. The population structure is a lattice network. Parameter values: N = 400, d = 5, β = 0.05, $\mathcal{C}=c=1$ for all games, except r_s = 3 in the PGG and TPGG, r_s = 5 and T = [d/2] + 2 in the dSH, and ${\mathcal{B}}_s=12$ in the dSD.

4. Discussion

In natural populations, the biotic and abiotic environment that organisms are exposed to varies persistently in time and space. To win the struggle for survival in this uncertain world, organisms have to timely adjust their behaviours in response to the fluctuation of their living environments [25,39]. For the longstanding conundrum of how cooperation can evolve, however, the majority of the existing evolutionary interpretations has been devoted to understanding the static interactive scenarios [1,6]. Therefore, when individual interactions, especially involving multiple players at a time, occur in the changing environment, determining whether cooperation can evolve will become fairly tricky. Here, we developed a general model framework by introducing the adaptation mechanism of reinforcement learning to investigate how cooperation can evolve in the constantly changing multi-player game environment. Our model not only considers the interplay between players’ behaviours and environmental variations, but also incorporates a cognitive or psychological feedback loop where players’ choices determine the game outcome, and in turn are affected by it. Such a setting is, to some extent, analogous to the human decision in the context of the hybrid human–machine cooperation [65], a key research theme in the emerging interdisciplinary field of machine behaviour [66], in which humans can use algorithms to make decisions and subsequently the training of the same algorithms is affected by those decisions.

The importance of environmental variations in population dynamics has long been recognized in theoretical ecology and population biology [24–26]. In a realistic social or ecological system, individual behaviours and environmental variations are inevitably coupled together [24,25]. By consuming, transforming, or producing common-pool resources, for example, organisms are enabled to alter their living environments, and consequently, such modification may consequentially be detrimental or beneficial to their survival [22]. Our analytical condition for determining whether cooperation can be favoured over defection indeed provides us a plausible theoretical explanation for this phenomenon. If mutual actions of individuals lead the environment to transit from a preferable state where cooperation is more profitable to a hostile one where defection is more dominant, cooperation will be suppressed. By contrast, cooperation will flourish if the transition order is reversed. In particular, if the population has access to switching among multiple environmental states, the environment will play the role of intermediates in social interactions and the final outcome of whether cooperation can evolve will be the synthesis of results in each environmental state. Such an observation is different from the recent findings where game transitions can result in a more favourable outcome for cooperation even if all individual games favour defection [21,29]. One important reason for this is that we do not follow the scheme to explicitly assign a specific rule to prescribe the update of environmental states (i.e. model-specific methods), but rather simply assuming the ergodicity for the environmental dynamics (i.e. model-free methods). Thus, in this sense, our model is general and can be applied to a large variety of environmental dynamic processes.

Moreover, compared with the existing studies on the evolution of cooperation in the changing environment [20,21,23,27–29], another striking difference is that, apart from the environmental feedback, our model introduces the learning mechanism of reinforcement. Since, when the environment changes, the previous decision-making scheme adopted by individuals may fail to work, they must learn how to adjust their behaviours in response to the contingencies given by the environment, in order to obtain a higher fitness. Such a scenario is also closely related to some recent work across disciplines, including complexity science [49,67–70], artificial intelligence [44,56,71], evolutionary biology [72,73] and neuroscience [43]. However, their dominant attention has been paid to learning dynamics, the deterministic limit of the learning process, the design of new learning algorithms in games, or neural computations. In comparison, our model is discrete and stochastic, and focuses on multi-player stochastic games. In particular, our analysis for the game system is systematic and encompasses a variety of factors, such as group interactions, spatial structures and environmental variations. In addition, our work may offer some new insight into the interface between reinforcement learning and evolutionary game theory from the perspective of function approximation [44,50], because most existing progress in combining tools from these two fields to explore the interaction of multiple agents is based on value-based methods [49,56,70,71].

In the present work, one of the main limitations is that the strategic update is restricted to the asynchronous type and the learning experience is required to be shared among individuals. Although such a setting is appropriate in those scenarios where individuals modify their strategies independently, and typical in economics applications and for overlapping generations [11], it has been suggested that the unanimous satisfactory decisions reached by all asynchronous update individuals cannot always be guaranteed by synchronous updates [74]. In particular, if individuals are able to communicate with each other via a network or leverage the perceived information to model and infer the choices of others [45,47], the asynchronous update will suffer from some difficulties. Thus, further work on synchronously strategic revisions is worth exploring in the future. Of course, such an extension will also be full of challenges, because updating strategies concurrently for multiple agents will inevitably give rise to some complications, such as the curse of dimensionality, requirement for coordination, non-stationarity and exploration–exploitation trade-off [45]. Moreover, some further efforts should be invested in the partial observability of the Markov environmental states and relaxing the perfect environmental information required in our model to the unobservable or unpredictable type [75].

Data accessibility

This article has no additional data.

Authors' contributions

F.H., M.C. and L.W. participated in the design of the study and drafted the manuscript.

Competing interests

We declare we have no competing interest.

Funding

This work was supported by the National Natural Science Foundation of China (grant nos. 61751301 and 61533001). F.H. acknowledges the support from China Scholarship Council (grant no. 201906010075). M.C. was supported in part by the European Research Council (grant no. ERC-CoG-771687) and The Netherlands Organization for Scientific Research (grant no. NWO-vidi-14134).