On the complexity of computing Markov perfect equilibrium in general-sum stochastic games

ABSTRACT

Similar to the role of Markov decision processes in reinforcement learning, Markov games (also called stochastic games) lay down the foundation for the study of multi-agent reinforcement learning and sequential agent interactions. We introduce approximate Markov perfect equilibrium as a solution to the computational problem of finite-state stochastic games repeated in the infinite horizon and prove its PPAD-completeness. This solution concept preserves the Markov perfect property and opens up the possibility for the success of multi-agent reinforcement learning algorithms on static two-player games to be extended to multi-agent dynamic games, expanding the reign of the PPAD-complete class.

We proved that solving Markov perfect equilibria of general-sum stochastic games is PPAD-complete, which lays down the foundation of algorithmic complexity for multi-agent reinforcement learning methodology.

INTRODUCTION

Shapley [1] introduced stochastic games (SGs) to study the dynamic non-cooperative multi-player game, where each player simultaneously and independently chooses an action at each round for a reward. According to their current state and the chosen actions, the next state is determined by a probability distribution specified a priori. Shapley’s work includes the first proof of the existence of a stationary strategy profile under which no agent has an incentive to deviate, in two-player zero-sum SGs. Next, the existence of equilibrium in stationary strategies was extended to multi-player general-sum SGs by Fink [2]. Such a solution concept (known as Markov perfect equilibrium (MPE) [3]) captures the dynamics of multi-player games.

Because of its generality, the framework of SGs has enlightened a sequence of studies [4] on a wide range of real-world applications ranging from advertising and pricing [5], species interaction game modeling in fisheries [6], traveling inspection [7] and gaming AIs [8]. As a result, developing algorithms to compute MPE in SGs has become one of the key subjects in this extremely rich research domain, using approaches from applied mathematics, economics, operations research, computer science and artificial intelligence (see, e.g., [9]).

The concept of the SG underpins many AI and machine learning studies. The optimal policy making of Markov decision processes (MDPs) captures the central problem of a single agent interacting with its environment, according to Sutton and Barto [10]. In multi-agent reinforcement learning (MARL) [11,12], SG extends to incorporate the dynamic nature in multi-agent strategic interactions, to study optimal decision making and subsequently equilibria in multi-player games [13,14].

For two-player zero-sum (discounted) SGs, the game-theoretical equilibrium is closely related to the optimization problem in MDPs as the opponent is purely adversarial [15,16]. On the other hand, solving general-sum SGs has been possible only under strong assumptions [2,17]. Zinkevich et al. [18] demonstrated that, for the entire class of value iteration methods, it is difficult to find stationary Nash equilibrium (NE) policies in general-sum SGs. This has led to few existing MARL algorithms to general-sum SGs. Known approaches have either studied special cases of SGs [19,20] or ignored the dynamic nature to limit the study to the weaker notion of Nash equilibrium [21].

Recently, Solan and Vieille [22] reconfirmed the importance of the existence of a stationary strategy profile as having several philosophical implications. First, it is conceptually straightforward. Second, past play affects the players’ future behavior only through the current state. Third, subsequently and most importantly, equilibrium behavior does not involve non-credible threats, a property that is stronger than the equilibrium property, and viewed as highly desirable [23].

Surprisingly, the complexity of finding an MPE in an SG remains an open problem, although an SG was proposed more than sixty years ago and despite its importance. While fruitful studies have been conducted on zero-sum SGs, we still know little about the complexity of solving general-sum SGs. It is clear that solving MPE in (infinite-horizon) SGs is at least PPAD-hard, since solving a two-player NE in one-shot SGs is already complete in this computational class [24,25] defined by Papadimitriou [26]. This suggests that it is unlikely to have polynomial-time algorithms in general-sum stochastic games for two players. Yet, with complications involved in the general-sum and dynamic settings, the unresolved challenge has been: Can solving MPE in general-sum SGs be anywhere complete in computational complexity classes?

We answer the above question in the positive, proving the computation of an approximate MPE in SGs equivalent to that of a Nash equilibrium in a single state setting, and subsequently showing its PPAD-completeness. It opens up the possibility to develop MARL algorithms to work for the general-sum SGs in the same way as for an ordinary Nash equilibrium computation.

Intuitions and a sketch of our main ideas

Computational studies on problem solving build understanding on various types of reduction. After all, computations carried out on computers are eventually reduced to AND/OR/NOT gates on electronic circuits.

To prove that a problem is PPAD-complete, we need to prove that it is in the class, and that it can be used as a base to solve any other problem in this class (for its hardness). More formally, the reduction needs to be carried out in polynomial (with respect to the input size) time. Nash equilibrium computation of two-player normal-form games [27] is arguably the most prominent PPAD-complete problem [24,25]. When one stochastic game has only one state and the discount factor γ = 0, then finding an MPE is equivalent to finding a Nash equilibrium in the corresponding normal-form game. The PPAD-hardness of finding an MPE follows immediately. Our main result is to prove the PPAD membership property of computing an approximate MPE (Lemma 2 below).

Firstly, we construct a function f on the strategy profile space, such that each strategy profile is a fixed point of f if and only if it is an MPE of the stochastic game (Theorem 2 below). Furthermore, we prove that the function f is continuous (λ-Lipschitz by Lemma 3 below), so that fixed points are guaranteed to exist by the Brouwer fixed point theorem.

Secondly, we prove that the function f has some ‘good’ approximation properties. Let be the input size of a stochastic game. If we can find a -approximate fixed point π of f, i.e. , where π is a strategy profile, then π is an ε-approximate MPE for the stochastic game (combining Lemma 5 and Lemma 6 below). So our goal converts to finding an approximate fixed point of a Lipschitz function.

Finally, our PPAD membership follows from the theorem that computation of the approximate Brouwer fixed point of a Lipschitz function is PPAD-complete, as shown in the seminal paper by Papadimitriou [26].

Related work

In practice, MARL methods are most often applied to compute the MPE of an SG based on the interactions between agents and the environment. Their uses can be classified in two different settings: online and offline. In the offline setting (also known as the batch setting [21]), the learning algorithm controls all players in a centralized way, hoping that the learning dynamics can eventually lead to an MPE by using a limited number of interaction samples. In the online setting, the learner controls only one of the players to play with an arbitrary group of opponents in the game, assuming unlimited access to the game environment. The central focus is often on the regret: the difference between the learner’s total reward during learning versus that of a benchmark measure in hindsight.

In the offline setting, two-player zero-sum (discounted) SGs have been extensively studied. Since the opponent is purely adversarial in zero-sum SGs, the process of seeking the worst-case optimality for each player can be thought of as solving MDPs. As a result, (approximate) dynamic programming methods [28,29] such as least-squares policy iteration [30] and fitted value iteration [31] or neural fitted Q iteration [32] can be adopted to solve SGs [33–36]. Under this setting, policy-based methods [37,38] can also be applied. However, directly applying existing MDP solvers on general-sum SGs is problematic. Since solving two-player NE in general-sum normal-form games (i.e. one-shot SGs) is well known to be PPAD-complete [24,25], the complexity of MPE in general-sum SGs is expected to be at least PPAD-hard. Although early attempts such as Nash-Q learning [39], correlated-Q learning [40], friend-or-foe Q-learning [41] have been made to solve general-sum SGs under strong assumptions, Zinkevich et al. [18] demonstrated that none in the entire class of value iteration methods can find stationary NE policies in general-sum SGs. The difficulties on both the complexity side and the algorithmic side have led to few existing MARL algorithms for general-sum SGs. Successful approaches either assume knowing the complete information of the SG such that solving MPE can be turned into an optimization problem [42], or prove the convergence of batch RL methods to a weaker notion of NE [21].

In the online setting, agents aim to minimize their regret by trial and error. One of the most well-known online algorithms is R-MAX [43], which studies (average-reward) zero-sum SGs and provides a polynomial (in game size and error parameter) regret bound while competing against an arbitrary opponent. Following the same regret definition, UCSG [44] improved R-MAX and achieved a sublinear regret, but still in the two-player zero-sum SG setting. When it comes to MARL solutions, Littman [13] proposed a practical solution named Minimax-Q that replaces the max operator with the minimax operator. Asymptotic convergence results of Minimax-Q were developed in both tabular cases [45] and value function approximations [46]. To avoid the overly pessimism property by playing the minimax value for general-sum SGs, WoLF [47] was proposed to take variable steps to exploit an opponent’s suboptimal policy for a higher reward on a variety of stochastic games. AWESOME [48] further generalized WoLF and achieved NE convergence in multi-player general-sum repeated games. However, outside the scope of zero-sum SGs, the question [43] of whether a polynomial time no-regret (near-optimal) MARL algorithm exists for general-sum SGs remains open.

Some recent works studied the sample complexity issue in RL and MARL algorithms, most of which considered a finite horizon. Jin et al. [49] proved that a variant of Q-learning with upper confidence bound exploration can achieve a near-optimal sample efficiency under episodic MDP setting. Zhang et al. [50] proposed a learning algorithm for episodic MDP with a regret bound close to the information theoretic lower bound. Li et al. [51] proposed a probably approximately correct learning algorithm for episodic RL with a sample complexity independent of the planning horizon. For general-sum MARL, Chen et al. [52] proved an exponential lower bound on the sample complexity of approximate Nash equilibrium even in n-player normal-form games. In the same direction, Song et al. [53] showed that correlated equilibrium (CE) and coarse correlated equilibrium (CCE) can be learned within a sample complexity polynomial in the maximum size of the action set of a player, rather than the size of the joint action space. Jin et al. [54] developed a decentralized MARL algorithm with polynomial sample complexity to learn CE and CCE.

DEFINITIONS AND THE MAIN THEOREM

Definition 1 (Stochastic game). A stochastic game is defined by a tuple of six elements .

• By n we denote the number of agents.

• By we denote the set of finite environmental states. Let .

• By we denote the action space of agent i. Note that each agent i can choose different actions under different states. Without loss of generality, we assume that, for each agent i, the action space under each state is the same. Here is the set of agents’ joint action vector. Let and A_max = max _{i ∈ [n]}Aⁱ.

• By we denote the transition probability, that is, at each time step, given the agents’ joint action vector , then the transition probability from state s to state s′ in the next time step is P(s′|s, a).

• By we denote the reward function, that is, when agents are at state s and play the joint action vector a, then agent i will get reward rⁱ(s, a). We assume that the rewards are bounded by r_max.

• By γ ∈ [0, 1) we denote the discount factor that specifies the degree to which the agent’s rewards are discounted over time.

Each agent aims to find a behavioral strategy with Markovian property, which is conditioned on the current state of the game.

The pure strategy space of agent i is , which means that agent i needs to select an action at each state. Note that the size of the pure strategy space of each agent is , which is already exponential in the number of states. More generally, we define the mixed behavioral strategy as follows.

Definition 2 (Behavioral strategy). A behavioral strategy of agent i is . For all , πⁱ(s) is a probability distribution on .

In the rest of the paper, we focus on behavioral strategy and refer to it simply as a strategy for convenience. A strategy profile π is the Cartesian product of all agents’ strategies, i.e. π = π¹ × ⋅⋅⋅ × πⁿ. We denote the probability of agent i using action aⁱ at state s by πⁱ(s, aⁱ). The strategy profile of all the agents other than agent i is denoted by π⁻ⁱ. We use πⁱ, π⁻ⁱ to represent π, and aⁱ, a⁻ⁱ to represent a.

Given π, the transition probability and the reward function depend only on the current state . Let r^{i, π}(s) denote and P^π(s′|s) denote . Fix π⁻ⁱ; the transition probability and the reward function depend only on the current state and player i’s action aⁱ. Let denote and denote .

For any positive integer m, let . Define . Then, for all , and .

Definition 3 (Value function). A value function for agent i under strategy profile π, , gives the expected sum of its discounted rewards when the starting state is s:

Here, s₀, s₁, … is the Markov chain such that the transition matrix is P^π, that is, for all k = 0, 1, …. Equivalently, the value function can be defined recursively via the Bellman policy equation 

Definition 4 (Markov perfect equilibrium). A behavioral strategy profile π is called a Markov perfect equilibrium if, for all , all i ∈ [n] and all ,

where is the value function of i when its strategy deviates to while the strategy profile of other agents is π⁻ⁱ.

The Markov perfect equilibrium is a solution concept within SGs in which the players’ strategies depend only on the current state but not on the game history.

Definition 5 (ε-approximate MPE). Given ε > 0, a behavioral strategy profile π is called an ε-approximate MPE if, for all , all i ∈ [n] and all ,

We use Approximate MPE to denote the computational problem of finding an approximate Markov perfect equilibrium in stochastic games, where the inputs and outputs are as follows. The input instance of problem Approximate MPE is a pair , where is a stochastic game and L is a positive integer. The output of problem Approximate MPE is a strategy profile , also dependent only at the current state but not on its history, such that π is a 1/L-approximate MPE of . We use the notation to denote the input size of the stochastic game .

Theorem 1 (Main theorem). Approximate MPEis PPAD-complete.

We note that, when |S| = 1 and γ = 0, a stochastic game degenerates to an n-player normal-form game. At this time, any MPE of this stochastic game is a Nash equilibrium for the corresponding normal-form game. So we have the following hardness result immediately.

Lemma 1. Approximate MPE is PPAD-hard.

To derive Theorem 1, we focus on the proof of PPAD membership of Approximate MPE in the rest of the paper.

Lemma 2. Approximate MPE is in PPAD.

ON THE EXISTENCE OF MPE

The original proof of the existence of MPE is from [2] and based on Kakutani’s fixed point theorem. Unfortunately, proofs that are based on Kakutani’s fixed point theorem in general cannot be turned into PPAD-membership results. We develop a proof that uses Brouwer’s fixed point theorem, based on which we also prove the PPAD membership of Approximate MPE.

Inspired by the continuous transformation defined by Nash to prove the existence of the equilibrium point [27], we define an updating function to adjust the strategy profile of agents in a stochastic game to establish the existence of MPE.

Let be the behavioral strategy profile under discussion.

Let denote the expected sum of discounted rewards of agent i if agent i uses pure action aⁱ at state s at the first step, and then follows πⁱ after that, but every other agent j maintains its strategy π^j. Formally,

For each player i ∈ [n] with each action at each state , we define a policy update function of πⁱ(s, aⁱ) as

We consider the infinite norm distance of two strategy profiles π₁ and π₂, denoted by ‖π₁ − π₂‖_∞: .

We first prove that the function f satisfies a continuity property, namely, is λ-Lipschitz for λ equal to .

Lemma 3. The function f is λ-Lipschitz, i.e. for everysuch that ‖π₁ − π₂‖_∞ ≤ δ, we have

Proof. At any , pick any player i ∈ [n]. For an action , let M₁(aⁱ) denote and . From the next claim (proof in the Appendix),

Claim 1. For any x, x′, y, y′, z, z′ ≥ 0 such that (x + y)/(1 + z) ≤ 1 and (x′ + y′)/(1 + z′) ≤ 1, it holds that

Take δ = ‖π₁ − π₂‖_∞; then for any . Next, for any , we estimate

We should first derive an upper bound on .

Claim 2. It holds that

This follows from the following claim (proof in the Appendix).

Claim 3. It holds that

Similarly, we have the following claim.

Claim 4. It holds that

To bound for every , we denote by the column vector , and by r^{i, π} the column vector and by P^π the matrix . By the Bellman policy equation (Definition 3), we have

which means that

We prove in Lemma 7 below that

for all .

Now we are ready to give an upper bound on for any . We have

where the forth line follows from , in Lemma 7 below.

Similarly, we establish a bound for :

For any , we have

Thus, for any and any , we obtain

This completes the proof of Lemma 3.□

Now we can establish the existence of MPE by the Brouwer fixed point theorem.

Theorem 2. For any stochastic game , a strategy profile π is an MPE if and only if it is a fixed point of the function f, i.e. f(π) = π. Furthermore, the function f has at least one fixed point.

Proof. We first show that the function f has at least one fixed point. Brouwer’s fixed point theorem states that, for any continuous function mapping a compact convex set to itself, there is a fixed point. Note that f is a function mapping a compact convex set to itself. Also, f is continuous by Lemma 3. Therefore, the function f has at least one fixed point.

We then prove that a strategy profile π is an MPE if and only if it is a fixed point of f.

⇒: For the necessity, suppose that π is an MPE; then, by Definition 4, we have, for each player i ∈ [n], each state and each policy , . By Lemma 4 to be proven next, we have, for any action , , which implies that . Then, for each player i ∈ [n], each state and each action , (f(π))ⁱ(s, aⁱ) = πⁱ(s, aⁱ). It follows that π is a fixed point of f.

⇐: For the proof of the sufficiency part, let π be a fixed point of f. Then, for each player i ∈ [n], each state and each action ,

We first provide the following claim given the condition that π is a fixed point.

Claim 5. For any , .

Proof of Claim 5. Suppose for contradiction that there exists i ∈ [n] and such that . The above fixed point equation implies that πⁱ(s, dⁱ) > 0.

Let ; then . Note that, by the recursive definition of , we have

Since , there must exist some such that , because otherwise we have for all , which, com-bined with , implies that , a contradiction to the above equation. With some further calculation, we can have the equation

The above strict inequality follows because as well as πⁱ(s, cⁱ) > 0.

This contradicts with the assumption that π is a fixed point of f. Therefore, it holds for any that .□

Combining Claim 5 and Lemma 4 (to be proven next), we find that, for any and any , . Thus, π is an MPE by definition. This completes the proof of Theorem 2.□

Lemma 4. For any player i ∈ [n], given π⁻ⁱ, for any, the following two statements are equivalent:

1. for all and all , ;

2. for all and all , .

Proof. Let denote the space of value functions , and define the l_∞ norm for any as .

Pick any player i ∈ [n] and keep π⁻ⁱ fixed. Define the Bellman operator such that, for any and any ,

Note that, for all , , since .

We first prove the equivalence between statements 1 and 2, based on Claim 6 below, which will be proved next for completeness.

2⇒1: From statement 2, for all , , which is the fixed point of Φⁱ by Claim 6 below. That is, for all , , by definition of the Bellman operator Φⁱ. Statement 1 holds.

1⇒2: If statement 1 holds, we have, for all , . Since by Claim 6 below, we get . , implying that is a fixed point of Φⁱ. By Claim 6, the unique fixed point of Φⁱ is . Therefore, for all , : statement 2 holds.

Claim 6. We have the following important properties.

• It holds that Φⁱis a γ-contraction mapping with respect to the l_∞norm, and has a unique fixed point.

• For any and any , .

• Let denote the fixed point of Φⁱ; then vⁱ* is the optimal value function, i.e. for any, .

Proof of Claim 6. Define

We have for all and .

We first prove that Φⁱ is a γ-contraction mapping with respect to the l_∞ norm. For all , let . We show that ‖Φⁱ(v₁) − Φⁱ(v₂)‖_∞ ≤ γδ.

For all and all , observe that

so .

Without loss of generality, one can suppose that Φⁱ(v₁)(s) ≥ Φⁱ(v₂)(s). Taking arbitrary , we have

thus, |Φⁱ(v₁)(s) − Φⁱ(v₂)(s)| ≤ γδ. By symmetry, the claim holds for the case in which Φⁱ(v₁)(s) ≤ Φⁱ(v₂)(s). Therefore, it holds that ‖Φⁱ(v₁) − Φⁱ(v₂)‖_∞ ≤ γδ. Thus, Φⁱ is a γ-contraction mapping.

By the Banach fixed point theorem, we know that has a unique fixed point . Moreover, for any , the point sequence v, Φⁱ(v), Φⁱ(Φⁱ(v)), … converges to vⁱ*, i.e. for all , lim_{k → ∞}(Φⁱ)^(k)(v)(s) = vⁱ*(s), where (Φⁱ)^(k) = Φⁱ (Φⁱ)^{(k − 1)} is defined recursively with (Φⁱ)⁽¹⁾ = Φⁱ.

Next, for all and all , , since

by definition.

Finally we prove that, for any , . For any , define the operator , such that, for any and any ,

Note that, for any , is also a γ-contraction mapping. This is because, for any such that ‖v₁ − v₂‖_∞ = δ, we have shown that, for any and any , , so

and then .

For any , we can observe that by definition. By the Banach fixed point theorem, we know that has a unique fixed point in , so is exactly the unique fixed point of .

Now we arbitrarily take a policy such that, for all , . It can be seen that, for any,

It follows that , so vkⁱ* is a fixed point of . Since the unique fixed point of is , we have . Thus, for any , .

To show that, for any , , we observe that, given , if for all , then, for any and any , . Therefore, . As we have, for all and all , , we have, for any and any , by induction. It follows that . Let k → ∞; then we get . Thus, .

The claim that, for all , follows.

THE APPROXIMATION GUARANTEE

Theorem 2 states that π is a fixed point of f if and only if π is an MPE for the stochastic game. Now we prove that f has some good approximation properties: if we find an ε-approximate fixed point π of f then it is also a -approximate MPE for the stochastic game (combining the following Lemma 5 and Lemma 6). This implies the PPAD-membership of Approximate MPE.

Lemma 5. Let ε > 0 and π be a strategy profile. If ‖f(π) − π‖_∞ ≤ ε then, for each player i ∈ [n] and each state, we have

where

Proof. Pick any player i ∈ [n] and any state . For simplicity, for any , define and .

First we give an upper bound on M(aⁱ). For any , it can be easily seen that

By the condition ‖f(π) − π‖_∞ ≤ ε , for any , we have

Set ε′ = (1 + A_maxr_max/(1 − γ))ε; then we have the crucial inequality

(1)

Let denote or, equivalently, . Let .

Case 1: . By inequality (1) we have

Case 2: . By inequality (1) we have

(2)

As and, for all , ,

Therefore,

Moreover, observe that . Substituting these into inequality (2), we get

It follows that .

In conclusion, combining the two cases, we get

Thus, for each , we have

which completes the proof.□

Lemma 6. Let ε > 0 and π be a strategy profile. If, for each player i ∈ [n] and each state, then π is an ε/(1 − γ)-approximate MPE.

Proof. Recall the mapping , defined as the Bellman operator, from the proof of Lemma 4. Let be the unique fixed point of Φⁱ and recall that, for all , .

Pick any player i ∈ [n]; by assumption, for each state , we have . On the other hand, , so we have , i.e. .

Since Φⁱ is a γ-contraction mapping,

In addition, by the triangle inequality we have

so it follows that

Thus, we have

It follows that, for any and any ,

By definition, it follows that π is an ε/(1 − γ)-approximate MPE.□

To conclude, Lemma 2 is proven by combining Lemma 5 and Lemma 6, which completes the proof of Lemma 1.

CONCLUSION

Solving an MPE in general-sum SGs has long expected to be at least PPAD-hard. In this paper, we prove that computing an MPE in a finite-state infinite horizon discounted SG is PPAD-complete. Our completeness result also immediately implies the PPAD-completeness of computing an MPE in action-free SGs and single-controller SGs. We hope that our results can encourage MARL researchers to study solving an MPE in general-sum SGs, proposing a sample-efficient MARL solution, which leads to more prosperous algorithmic developments than those currently on zero-sum SGs.

APPENDIX

Proof of Claim 1

We have

The first and third inequalities follow by the triangle inequality, the second inequality holds because x′ + y′ ≤ 1 + z′ and the last inequality follows because 1 + z ≥ 1. It immediately follows that

Proof of Claim 2

We have

where the last inequality follows from the next claim.

Proof of Claim 3

We have

Proof of Claim 4

We have

Lemma 7 and its proof

Lemma 7. For every such that ‖π₁ − π₂‖_∞ ≤ δ, we have

for any.

Proof. We first give an upper bound on for any :

Now we view P^π as an S × S matrix. For any two S × S matrices M¹, M², we use ‖M¹ − M²‖_max to denote max _{i, j}|M¹(i, j) − M²(i, j)|, i.e. the max norm. Then we have .

Let and . (Note that the inverse of (I − γP^π) must exist because γ < 1.)

By definition, we have and . Then

where the sixth line follows the following facts:

1. ;

2. |Q¹(k, j) − Q²(k, j)| ≤ max _k|Q¹(k, j) − Q²(k, j)|;

3. ;

4. .

Note that . Since the 1-norm is submultiplicative, we have

which leads to the fourth fact. So we have

This completes the proof.

FUNDING

This work was partially supported by the Science and Technology Innovation 2030—‘New Generation of Artificial Intelligence’ Major Project (2018AAA0100901).

AUTHOR CONTRIBUTIONS

X.D., D.M. and J.W. designed the research; X.D. and D.M. identified the research problem; X.D. and N.L. performed the research; D.M. and Y.Y. coordinated the team; X.D., N.L. and Y.Y. wrote the paper.

Conflict of interest statement. None declared.