Figure 1.

Illustration of evolutionary dynamics for 4-player stochastic games in the structured population. (a) At a time step t, a random individual is chosen as the focal player (depicted by the dashed red circle), and then three of its neighbours are selected randomly as co-players to form a 4-player game (because the focal player only has three neighbours, all of them are chosen), which is depicted by the light magenta shaded area. Conditioned on the focal player’s action and the environmental state s_t at time t, the environmental state at time t + 1 will be changed to s_t+1 with a transition probability. Similarly, a new round of the game will be reorganized at time t + 1. This process is repeated infinitely. (b) At time t, after perceiving the environmental state s_t and the co-players’ configuration j, the focal player uses policy π to determine which action to be taken, whereas its co-players still use their previous actions in the past round. At the end of this round, each player will gain a payoff, which will play a role of the feedback signal and will assist the focal player to update its policy.