Forward and Backward Bellman Equations Improve the Efficiency of the EM Algorithm for DEC-POMDP

Abstract

Decentralized partially observable Markov decision process (DEC-POMDP) models sequential decision making problems by a team of agents. Since the planning of DEC-POMDP can be interpreted as the maximum likelihood estimation for the latent variable model, DEC-POMDP can be solved by the EM algorithm. However, in EM for DEC-POMDP, the forward–backward algorithm needs to be calculated up to the infinite horizon, which impairs the computational efficiency. In this paper, we propose the Bellman EM algorithm (BEM) and the modified Bellman EM algorithm (MBEM) by introducing the forward and backward Bellman equations into EM. BEM can be more efficient than EM because BEM calculates the forward and backward Bellman equations instead of the forward–backward algorithm up to the infinite horizon. However, BEM cannot always be more efficient than EM when the size of problems is large because BEM calculates an inverse matrix. We circumvent this shortcoming in MBEM by calculating the forward and backward Bellman equations without the inverse matrix. Our numerical experiments demonstrate that the convergence of MBEM is faster than that of EM.

1. Introduction

Markov decision process (MDP) models sequential decision making problems and has been used for planning and reinforcement learning [1,2,3,4]. MDP consists of an environment and an agent. The agent observes the state of the environment and controls it by taking actions. The planning of MDP is to find the optimal control policy maximizing the objective function, which is typically solved by the Bellman equation-based algorithms such as value iteration and policy iteration [1,2,3,4].

Decentralized partially observable MDP (DEC-POMDP) is an extension of MDP to a multiagent and partially observable setting, which models sequential decision making problems by a team of agents [5,6,7]. DEC-POMDP consists of an environment and multiple agents, and the agents cannot observe the state of the environment and the actions of the other agents completely. The agents infer the environmental state and the other agents’ actions from their observation histories and control them by taking actions. The planning of DEC-POMDP is to find not only the optimal control policy but also the optimal inference policy for each agent, which maximize the objective function [5,6,7]. Applications of DEC-POMDP include planetary exploration by a team of rovers [8], target tracking by a team of sensors [9], and information transmission by a team of devices [10]. Since the agents cannot observe the environmental state and the other agents’ actions completely, it is difficult to extend the Bellman equation-based algorithms for MDP to DEC-POMDP straightforwardly [11,12,13,14,15].

DEC-POMDP can be solved using control as inference [16,17]. Control as inference is a framework to interpret a control problem as an inference problem by introducing auxiliary variables [18,19,20,21,22]. Although control as inference has several variants, Toussaint and Storkey showed that the planning of MDP can be interpreted as the maximum likelihood estimation for a latent variable model [18]. Thus, the planning of MDP can be solved by EM algorithm, which is the typical algorithm for the maximum likelihood estimation of latent variable models [23]. Since the EM algorithm is more general than the Bellman equation-based algorithms, it can be straightforwardly extended to POMDP [24,25] and DEC-POMDP [16,17]. The computational efficiency of the EM algorithm for DEC-POMDP is comparable to that of other algorithms for DEC-POMDP [16,17,26,27,28], and the extensions to the average reward setting and to the reinforcement learning setting have been studied [29,30,31].

However, the EM algorithm for DEC-POMDP is not efficient enough to be applied to real-world problems, which often have a large number of agents or a large size of an environment. Therefore, there are several studies in which improvement of the computational efficiency of the EM algorithm for DEC-POMDP was attempted [26,27]. Because these studies achieve improvements by restricting possible interactions between agents, their applicability is limited. Therefore, it is desirable to have improvement in the efficiency for more general DEC-POMDP problems.

In order to improve the computational efficiency of EM algorithm for general DEC-POMDP problems, there are two problems that need to be resolved. The first problem is the forward–backward algorithm up to the infinite horizon. The EM algorithm for DEC-POMDP uses the forward–backward algorithm, which has also been used in EM algorithm for hidden Markov models [23]. However, in the EM algorithm for DEC-POMDP, the forward–backward algorithm needs to be calculated up to the infinite horizon, which impairs the computational efficiency [32,33]. The second problem is the Bellman equation. The EM algorithm for DEC-POMDP does not use the Bellman equation, which plays a central role in the the planning and in the reinforcement learning for MDP [1,2,3,4]. Therefore, the EM algorithm for DEC-POMDP cannot use the advanced techniques based on the Bellman equation, which makes it possible to solve large-size problems [34,35,36].

In some previous studies, resolution of these problems was attempted by replacing the forward–backward algorithm up to the infinite horizon with the Bellman equation [32,33]. However, in these studies, the computational efficiency could not be improved completely. For example, Song et al. replaced the forward–backward algorithm with the Bellman equation and showed that their algorithm is more efficient than EM and other DEC-POMDP algorithms by the numerical experiments [32]. However, since a parameter dependency is overlooked in [32], their algorithm may not find the optimal policy under a general situation (see Appendix D for more details). Moreover, Kumar et al. showed that the forward–backward algorithm can be replaced by linear programming with the Bellman equation as a constraint [33]. However, their algorithm may be less efficient than the EM algorithm when the size of problems is large. Therefore, previous studies have not yet completely improved the computational efficiency of EM algorithm for DEC-POMDP.

In this paper, we propose more efficient algorithms for DEC-POMDP than EM algorithm by introducing the forward and backward Bellman equations into it. The backward Bellman equation corresponds to the traditional Bellman equation, which has been used in previous studies [32,33]. In contrast, the forward Bellman equation has not yet been used for the planning of DEC-POMDP explicitly. This equation is similar to that recently proposed in the offline reinforcement learning of MDP [37,38,39]. In the offline reinforcement learning of MDP, the forward Bellman equation is used to correct the difference between the data sampling policy and the policy to be evaluated. In the planning of DEC-POMDP, the forward Bellman equation plays the important role in inferring the environmental state.

We propose the Bellman EM algorithm (BEM) and the modified Bellman EM algorithm (MBEM) by replacing the forward–backward algorithm with the forward and backward Bellman equations. They are different in terms of how to solve the forward and backward Bellman equations. BEM solves the forward and backward Bellman equations by calculating an inverse matrix. BEM can be more efficient than EM because BEM does not calculate the forward–backward algorithm up to the infinite horizon. However, since BEM calculates the inverse matrix, it cannot always be more efficient than EM when the size of problems is large, which is the same problem as [33]. Actually, BEM is essentially the same as [33]. In the linear programming problem of [33], the number of variables is equal to that of constraints, which enables us to solve it only from the constraints without the optimization. Therefore, the algorithm in [33] becomes equivalent to BEM, and they suffers from the same problem as BEM.

This problem is addressed by MBEM. MBEM solves the forward and backward Bellman equations by applying the forward and backward Bellman operators to the arbitrary initial functions infinite times. Although MBEM needs to calculate the forward and backward Bellman operators infinite times, which is the same problem with EM, MBEM can evaluate approximation errors more tightly owing to the contractibility of these operators. It can also utilize the information of the previous iteration owing to the arbitrariness of the initial functions. These properties enable MBEM to be more efficient than EM. Moreover, MBEM resolves the drawback of BEM because MBEM does not calculate the inverse matrix. Therefore, MBEM can be more efficient than EM even when the size of problems is large. Our numerical experiments demonstrate that the convergence of MBEM is faster than that of EM regardless of the size of problems.

The paper is organized as follows: In Section 2, DEC-POMDP is formulated. In Section 3, the EM algorithm for DEC-POMDP, which was proposed in [16], is briefly reviewed. In Section 4, the forward and backward Bellman equations are derived, and the Bellman EM algorithm (BEM) is proposed. In Section 5, the forward and backward Bellman operators are defined, and the modified Bellman EM algorithm (MBEM) is proposed. In Section 6, EM, BEM, and MBEM are summarized and compared. In Section 7, the performances of EM, BEM, and MBEM are compared through the numerical experiment. In Section 8, this paper is concluded, and future works are discussed.

2. DEC-POMDP

DEC-POMDP consists of an environment and N agents (Figure 1 and Figure 2a) [7,16]. $x_t\in \mathcal{X}$ is the state of the environment at time t. $y_t^i\in {\mathcal{Y}}^i$, $z_t^i\in {\mathcal{Z}}^i$, and $a_t^i\in {\mathcal{A}}^i$ are the observation, the memory, and the action available to the agent $i\in \{1,\dots ,N\}$, respectively. $\mathcal{X}$, ${\mathcal{Y}}^i$, ${\mathcal{Z}}^i$, and ${\mathcal{A}}^i$ are finite sets. ${\mathit{y}}_t:=(y_t^1,..,y_t^N)$, ${\mathit{z}}_t:=(z_t^1,..,z_t^N)$, and ${\mathit{a}}_t:=(a_t^1,..,a_t^N)$ are the joint observation, the joint memory, and the joint action of the N agents, respectively.

Figure 1.

Schematic diagram of DEC-POMDP. DEC-POMDP consists of an environment and N agents ($N=2$ in this figure). $x_t$ is the state of the environment at time t. $y_t^i$, $z_t^i$, and $a_t^i$ are the observation, the memory, and the action available to the agent $i\in \{1,\dots ,N\}$, respectively. The agents update their memories based on their observations, and take their actions based on their memories to control the environmental state. The planning of DEC-POMDP is to find their optimal memory updates and action selections that maximize the objective function.

Figure 2.

Dynamic Bayesian networks of DEC-POMDP (a) and the latent variable model for the time horizon $T\in \{0,1,2,\dots \}$ (b). $x_t$ is the state of the environment at time t. $y_t^i$, $z_t^i$, and $a_t^i$ are the observation, the memory, and the action available to the agent $i\in \{1,\dots ,N\}$, respectively. (a) $r_t\in \mathbb{R}$ is the reward, which is generated at each time. (b) $o\in \{0,1\}$ is the optimal variable, which is generated only at the time horizon T.

The time evolution of the environmental state $x_t$ is given by the initial state probability $p\left(x_0\right)$ and the state transition probability $p\left(x_{t+1}\right|x_t,{\mathit{a}}_t)$. Thus, agents can control the environmental state $x_{t+1}$ by taking appropriate actions ${\mathit{a}}_t$. The agent i cannot observe the environmental state $x_t$ and the joint action ${\mathit{a}}_{t-1}$ completely, and obtains the observation $y_t^i$ instead of them. Thus, the observation ${\mathit{y}}_t$ obeys the observation probability $p\left({\mathit{y}}_t\right|x_t,{\mathit{a}}_{t-1})$. The agent i updates its memory from $z_{t-1}^i$ to $z_t^i$ based on the observation $y_t^i$. Thus, the memory $z_t^i$ obeys the initial memory probability ${\nu }^i\left(z_0^i\right)$ and the memory transition probability ${\lambda }^i\left(z_t^i\right|z_{t-1}^i,y_t^i)$. The agent i takes the action $a_t^i$ based on the memory $z_t^i$ by following the action probability ${\pi }^i\left(a_t^i\right|z_t^i)$. The reward function $r(x_t,{\mathit{a}}_t)$ defines the amount of reward that is obtained at each step depending on the state of the environment $x_t$ and the joint action $a_t$ taken by the agents.

The objective function in the planning of DEC-POMDP is given by the expected return, which is the expected discounted cumulative reward:

$$\begin{array}{c}\hfill J\left(\mathit{\theta }\right)={\mathbb{E}}_{\mathit{\theta }}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(x_t,{\mathit{a}}_t)\right].\end{array}$$ (1)

$\mathit{\theta }:=(\mathit{\pi },\mathit{\lambda },\mathit{\nu })$ is the policy, where $\mathit{\pi }:=({\pi }^1,\dots ,{\pi }^N)$, $\mathit{\lambda }:=({\lambda }^1,\dots ,{\lambda }^N)$, and $\mathit{\nu }:=({\nu }^1,\dots ,{\nu }^N)$. $\gamma \in (0,1)$ is the discount factor, which decreases the weight of the future reward. The closer $\gamma$ is to 1, the closer the weight of the future reward is to that of the current reward.

The planning of DEC-POMDP is to find the policy $\mathit{\theta }$ that maximizes the expected return $J\left(\mathit{\theta }\right)$ as follows:

$$\begin{array}{c}\hfill {\mathit{\theta }}^{\ast }:=arg\underset{\mathit{\theta }}{max}J\left(\mathit{\theta }\right).\end{array}$$ (2)

In other words, the planning of DEC-POMDP is to find how to take the action and how to update the memory for each agent to maximize the expected return.

3. EM Algorithm for DEC-POMDP

In this section, we explain the EM algorithm for DEC-POMDP, which was proposed in [16].

3.1. Control as Inference

In this subsection, we show that the planning of DEC-POMDP can be interpreted as the maximum likelihood estimation for a latent variable model (Figure 2b).

We introduce two auxiliary random variables: the time horizon $T\in \{0,1,2,\dots \}$ and the optimal variable $o\in \{0,1\}$. These variables obey the following probabilities:

$$\begin{array}{cc}\hfill p\left(T\right)& =(1-\gamma ){\gamma }^T,\hfill \end{array}$$ (3)

$$\begin{array}{cc}\hfill p(o=1|x_T,{\mathit{a}}_T)& =\overline{r}(x_T,{\mathit{a}}_T):=\frac{r(x_T,{\mathit{a}}_T)-r_{min}}{r_{max}-r_{min}}\hfill \end{array}$$ (4)

where $r_{max}$ and $r_{min}$ are the maximum and the minimum value of the reward function $r(x,\mathit{a})$, respectively. Thus, $\overline{r}(x,\mathit{a})\in [0,1]$ is satisfied.

By introducing these variables, DEC-POMDP changes from Figure 2a to Figure 2b. While Figure 2a considers the infinite time horizon, Figure 2b considers the finite time horizon T, which obeys Equation (3). Moreover, while the reward $r_t:=r(x_t,{\mathit{a}}_t)$ is generated at each time in Figure 2a, the optimal variable o is generated only at the time horizon T in Figure 2b.

Theorem 1

([16]). The expected return $J\left(\mathit{\theta }\right)$ in DEC-POMDP (Figure 2a) is linearly related to the likelihood $p(o=1;\mathit{\theta })$ in the latent variable model (Figure 2b) as follows:

$$\begin{array}{c}\hfill J\left(\mathit{\theta }\right)={(1-\gamma )}^{-1}\left[(r_{max}-r_{min})p(o=1;\mathit{\theta })+r_{min}\right].\end{array}$$ (5)

Note that o is the observable variable, and $x_{0:T}$, ${\mathit{y}}_{1:T}$, ${\mathit{z}}_{0:T}$, ${\mathit{a}}_{0:T}$, T are the latent variables.

Proof. 

See Appendix A.1. □

Therefore, the planning of DEC-POMDP is equivalent to the maximum likelihood estimation for the latent variable model as follows:

$$\begin{array}{c}\hfill {\mathit{\theta }}^{\ast }=arg\underset{\mathit{\theta }}{max}p(o=1;\mathit{\theta }).\end{array}$$ (6)

Intuitively, while the planning of DEC-POMDP is to find the policy which maximizes the reward, the maximum likelihood estimation for the latent variable model is to find the policy which maximizes the probability of the optimal variable. Since the probability of the optimal variable is proportional to the reward, the planning of DEC-POMDP is equivalent to the maximum likelihood estimation for the latent variable model.

3.2. EM Algorithm

Since the planning of DEC-POMDP can be interpreted as the maximum likelihood estimation for the latent variable model, it can be solved by the EM algorithm [16]. EM algorithm is the typical algorithm for the maximum likelihood estimation of latent variable models, which iterates two steps, E step and M step [23].

In the E step, we calculate the Q function, which is defined as follows:

$$\begin{array}{c}\hfill Q(\mathit{\theta };{\mathit{\theta }}_k):={\mathbb{E}}_{{\mathit{\theta }}_k}\left[\left.logp(o=1,x_{0:T},{\mathit{y}}_{0:T},{\mathit{z}}_{0:T},{\mathit{a}}_{0:T},T;\mathit{\theta })\right|o=1\right]\end{array}$$ (7)

where ${\mathit{\theta }}_k$ is the current estimator of the optimal policy.

In the M step, we update ${\mathit{\theta }}_k$ to ${\mathit{\theta }}_{k+1}$ by maximizing the Q function as follows:

$$\begin{array}{c}\hfill {\mathit{\theta }}_{k+1}:=arg\underset{\mathit{\theta }}{max}Q(\mathit{\theta };{\mathit{\theta }}_k).\end{array}$$ (8)

Since each iteration between the E step and the M step monotonically increases the likelihood $p(o=1;{\mathit{\theta }}_k)$, we can find ${\mathit{\theta }}^{\ast }$ that locally maximizes the likelihood $p(o=1;\mathit{\theta })$.

3.3. M Step

Proposition 1

([16]). In the EM algorithm for DEC-POMDP, Equation (8) can be calculated as follows:

$$\begin{array}{cc}\hfill & {\pi }_{k+1}^i\left(\mathit{a}\right|\mathit{z})=\frac{{\sum }_{{\mathit{a}}^{-i},{\mathit{z}}^{-i}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z}){\sum }_{x,x^{\prime },{\mathit{z}}^{\prime }}p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z},\mathit{a};{\mathit{\lambda }}_k)F(x,\mathit{z};{\mathit{\theta }}_k)\left(\overline{r}(x,\mathit{a})+\gamma V(x,\mathit{z};{\mathit{\theta }}_k)\right)}{{\sum }_{\mathit{a},\mathit{z}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z}){\sum }_{x,x^{\prime },{\mathit{z}}^{\prime }}p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z},\mathit{a};{\mathit{\lambda }}_k)F(x,\mathit{z};{\mathit{\theta }}_k)\left(\overline{r}(x,\mathit{a})+\gamma V(x,\mathit{z};{\mathit{\theta }}_k)\right)},\hfill \end{array}$$ (9)

$$\begin{array}{cc}\hfill & {\lambda }_{k+1}^i\left({\mathit{z}}^{\prime }\right|\mathit{z},{\mathit{y}}^{\prime })=\frac{{\sum }_{{\mathit{z}}^{-i^{\prime }},{\mathit{z}}^{-i},{\mathit{y}}^{-i^{\prime }}}{\mathit{\lambda }}_k\left({\mathit{z}}^{i^{\prime }}\right|{\mathit{z}}^i,{\mathit{y}}^{i^{\prime }}){\sum }_{x^{\prime },x}p(x^{\prime },{\mathit{y}}^{\prime }|x,\mathit{z};{\mathit{\pi }}_k)F(x,\mathit{z};{\mathit{\theta }}_k)V(x^{\prime },{\mathit{z}}^{\prime };{\mathit{\theta }}_k)}{{\sum }_{{\mathit{z}}^{{}^{\prime }},\mathit{z},{\mathit{y}}^{{}^{\prime }}}{\mathit{\lambda }}_k\left({\mathit{z}}^{i^{\prime }}\right|{\mathit{z}}^i,{\mathit{y}}^{i^{\prime }}){\sum }_{x^{\prime },x}p(x^{\prime },{\mathit{y}}^{\prime }|x,\mathit{z};{\mathit{\pi }}_k)F(x,\mathit{z};{\mathit{\theta }}_k)V(x^{\prime },{\mathit{z}}^{\prime };{\mathit{\theta }}_k)},\hfill \end{array}$$ (10)

$$\begin{array}{cc}\hfill & {\nu }_{k+1}^i\left(z\right)=\frac{{\sum }_{{\mathit{z}}^{-i}}{\mathit{\nu }}_k\left(\mathit{z}\right){\sum }_x{p}_0\left(x\right)V(x,\mathit{z};{\mathit{\theta }}_k)}{{\sum }_z{\mathit{\nu }}_k\left(\mathit{z}\right){\sum }_x{p}_0\left(x\right)V(x,\mathit{z};{\mathit{\theta }}_k)}.\hfill \end{array}$$ (11)

${\mathit{a}}^{-i}:=({\mathit{a}}^1,\dots ,{\mathit{a}}^{i-1},{\mathit{a}}^{i+1},\dots ,{\mathit{a}}^N)$. ${\mathit{y}}^{-i}$ and ${\mathit{z}}^{-i}$ are defined in the same way. $F(x,\mathit{z};\mathit{\theta })$ and $V(x,\mathit{z};\mathit{\theta })$ are defined as follows:

$$\begin{array}{cc}\hfill F(x,\mathit{z};\mathit{\theta })& :=\sum _{t=0}^{\infty }{\gamma }^t{p}_t(x,\mathit{z};\mathit{\theta }),\hfill \end{array}$$ (12)

$$\begin{array}{cc}\hfill V(x,\mathit{z};\mathit{\theta })& :=\sum _{t=0}^{\infty }{\gamma }^t{p}_0(o=1|x,\mathit{z},T=t;\mathit{\theta })\hfill \end{array}$$ (13)

where $p_t(x,\mathit{z};\mathit{\theta }):=p(x_t=x,{\mathit{z}}_t=\mathit{z};\mathit{\theta })$, and $p_0(o=1|x,z,T;\mathit{\theta }):=p(o=1|x_0=x,z_0=z,T;\mathit{\theta })$.

Proof. 

See Appendix A.2. □

$F(x,\mathit{z};\mathit{\theta })$ quantifies the frequency of the state x and the memory z, which is called the frequency function in this paper. $V(x,\mathit{z};\mathit{\theta })$ quantifies the probability of $o=1$ when the initial state and memory are x and z, respectively. Since the probability of $o=1$ is proportional to the reward, $V(x,\mathit{z};\mathit{\theta })$ is called the value function in this paper. Actually, $V(x,\mathit{z};\mathit{\theta })$ corresponds to the value function [33].

3.4. E Step

$F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ need to be obtained to calculate Equations (9)–(11). In [16], $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ are calculated by the forward–backward algorithm, which has been used in EM algorithm for the hidden Markov model [23].

In [16], the forward probability ${\alpha }_t(x,\mathit{z})$ and the backward probability ${\beta }_t(x,\mathit{z})$ are defined as follows:

$$\begin{array}{cc}\hfill {\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)& :=p_t(x,\mathit{z};{\mathit{\theta }}_k),\hfill \end{array}$$ (14)

$$\begin{array}{cc}\hfill {\beta }_t(x,\mathit{z};{\mathit{\theta }}_k)& :=p_0(o=1|x,\mathit{z},T=t;{\mathit{\theta }}_k).\hfill \end{array}$$ (15)

It is easy to calculate ${\alpha }_0(x,\mathit{z};{\mathit{\theta }}_k)$ and ${\beta }_0(x,\mathit{z};{\mathit{\theta }}_k)$ as follows:

$$\begin{array}{cc}\hfill {\alpha }_0(x,\mathit{z};{\mathit{\theta }}_k)& =p_0(x,\mathit{z};{\mathit{\nu }}_k):=p_0\left(x\right){\mathit{\nu }}_k\left(\mathit{z}\right),\hfill \end{array}$$ (16)

$$\begin{array}{cc}\hfill {\beta }_0(x,\mathit{z};{\mathit{\theta }}_k)& =\overline{r}(x,\mathit{z};{\mathit{\pi }}_k):=\sum _{\mathit{a}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})\overline{r}(x,\mathit{a}).\hfill \end{array}$$ (17)

Moreover, ${\alpha }_{t+1}(x,\mathit{z};{\mathit{\theta }}_k)$ and ${\beta }_{t+1}(x,\mathit{z};{\mathit{\theta }}_k)$ are easily calculated from ${\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)$ and ${\beta }_t(x,\mathit{z};{\mathit{\theta }}_k)$:

$$\begin{array}{cc}\hfill {\alpha }_{t+1}(x,\mathit{z};{\mathit{\theta }}_k)& =\sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x,\mathit{z}|x^{\prime },{\mathit{z}}^{\prime };{\mathit{\theta }}_k){\alpha }_t(x^{\prime },{\mathit{z}}^{\prime };{\mathit{\theta }}_k),\hfill \end{array}$$ (18)

$$\begin{array}{cc}\hfill {\beta }_{t+1}(x,\mathit{z};{\mathit{\theta }}_k)& =\sum _{x^{\prime },{\mathit{z}}^{\prime }}{\beta }_t(x^{\prime },{\mathit{z}}^{\prime };{\mathit{\theta }}_k)p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z};{\mathit{\theta }}_k)\hfill \end{array}$$ (19)

where

$$\begin{array}{cc}\hfill & p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z};{\mathit{\theta }}_k)=\sum _{{\mathit{y}}^{\prime },\mathit{a}}{\mathit{\lambda }}_k\left({\mathit{z}}^{\prime }\right|\mathit{z},{\mathit{y}}^{\prime })p\left({\mathit{y}}^{\prime }\right|x^{\prime },\mathit{a})p\left(x^{\prime }\right|x,\mathit{a}){\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z}).\hfill \end{array}$$ (20)

Equations (18) and (19) are called the forward and backward equations, respectively. Using Equations (16)–(19), ${\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)$ and ${\beta }_t(x,\mathit{z};{\mathit{\theta }}_k)$ can be efficiently calculated from $t=0$ to $t=\infty$, which is called the forward–backward algorithm [23].

By calculating the forward–backward algorithm from $t=0$ to $t=\infty$, $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ can be obtained as follows [16]:

$$\begin{array}{cc}\hfill F(x,\mathit{z};{\mathit{\theta }}_k)& =\sum _{t=0}^{\infty }{\gamma }^t{\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k),\hfill \end{array}$$ (21)

$$\begin{array}{cc}\hfill V(x,\mathit{z};{\mathit{\theta }}_k)& =\sum _{t=0}^{\infty }{\gamma }^t{\beta }_t(x,\mathit{z};{\mathit{\theta }}_k).\hfill \end{array}$$ (22)

However, $F(x,z;{\mathit{\theta }}_k)$ and $V(x,z;{\mathit{\theta }}_k)$ cannot be calculated exactly by this approach because it is practically impossible to calculate the forward–backward algorithm until $t=\infty$. Therefore, the forward–backward algorithm needs to be terminated at $t=T_{max}$, where $T_{max}$ is finite. In this case, $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ are approximated as follows:

$$\begin{array}{cc}\hfill F(x,\mathit{z};{\mathit{\theta }}_k)& =\sum _{t=0}^{\infty }{\gamma }^t{\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)\approx \sum _{t=0}^{T_{max}}{\gamma }^t{\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k),\hfill \end{array}$$ (23)

$$\begin{array}{cc}\hfill V(x,\mathit{z};{\mathit{\theta }}_k)& =\sum _{t=0}^{\infty }{\gamma }^t{\beta }_t(x,\mathit{z};{\mathit{\theta }}_k)\approx \sum _{t=0}^{T_{max}}{\gamma }^t{\beta }_t(x,\mathit{z};{\mathit{\theta }}_k).\hfill \end{array}$$ (24)

$T_{max}$ needs to be large enough to reduce the approximation errors. In the previous study, a heuristic termination condition was proposed as follows [16]:

$$\begin{array}{c}\hfill {\gamma }^{T_{max}}p(o=1|T_{max};{\mathit{\theta }}_k)\ll \sum _{T=0}^{T_{max}-1}{\gamma }^{T}p(o=1|T;{\mathit{\theta }}_k).\end{array}$$ (25)

$p(o=1|T;{\mathit{\theta }}_k)={\sum }_{x,\mathit{z}}{\alpha }_{T^{\prime }}(x,\mathit{z};{\mathit{\theta }}_k){\beta }_{T^{″}}(x,\mathit{z};{\mathit{\theta }}_k)$ where $T=T^{\prime }+T^{″}$. However, the relation between $T_{max}$ and the approximation errors is unclear in Equation (25). We propose a new termination condition to guarantee the approximation errors as follows:

Proposition 2.

We set an acceptable error bound $\epsilon >0$. If

$$\begin{array}{c}\hfill T_{max}>\frac{log(1-\gamma )\epsilon }{log\gamma }-1\end{array}$$ (26)

is satisfied, then

$$\begin{array}{cc}\hfill {∥F(x,\mathit{z};{\mathit{\theta }}_k)-\sum _{t=0}^{T_{max}}{\gamma }^t{\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)∥}_{\infty }& <\epsilon ,\hfill \end{array}$$ (27)

$$\begin{array}{cc}\hfill {∥V(x,\mathit{z};{\mathit{\theta }}_k)-\sum _{t=0}^{T_{max}}{\gamma }^t{\beta }_t(x,\mathit{z};{\mathit{\theta }}_k)∥}_{\infty }& <\epsilon \hfill \end{array}$$ (28)

are satisfied.

Proof. 

See Appendix A.3. □

3.5. Summary

In summary, the EM algorithm for DEC-POMDP is given by Algorithm 1. In the E step, we calculate ${\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)$ and ${\beta }_t(x,\mathit{z};{\mathit{\theta }}_k)$ from $t=0$ to $t=T_{max}$ by the forward–backward algorithm. In M step, we update ${\mathit{\theta }}_k$ to ${\mathit{\theta }}_{k+1}$ using Equations (9)–(11). The time complexities of the E step and the M step are ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^2{T}_{max})$ and ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^2\left|\mathcal{Y}\right|\left|\mathcal{A}\right|)$, respectively. Note that $\mathcal{A}:={\otimes }_{i=1}^N{\mathcal{A}}^i$, and $\mathcal{Y}$ and $\mathcal{Z}$ are defined in the same way. The EM algorithm for DEC-POMDP is less efficient when the discount factor $\gamma$ is closer to 1 or the acceptable error bound $\epsilon$ is smaller because $T_{max}$ needs to be larger in these cases.

Algorithm 1 EM algorithm for DEC-POMDP

$k\leftarrow 0$, Initialize ${\mathit{\theta }}_k$. $T_{max}\leftarrow \lceil (log(1-\gamma )\epsilon )/log\gamma -1\rceil$ while ${\mathit{\theta }}_k$ or $J\left({\mathit{\theta }}_k\right)$ do not converge do    Calculate $p(x^{\prime },z^{\prime }|x,z;{\mathit{\theta }}_k)$ by Equation (20).    //—E step—//     ${\alpha }_0(x,z;{\mathit{\theta }}_k)\leftarrow p_0(x,z;{\mathit{\nu }}_k)$     ${\beta }_0(x,z;{\mathit{\theta }}_k)\leftarrow \overline{r}(x,z;{\mathit{\pi }}_k)$    for  $t=1,2,\dots ,T_{max}$  do        ${\alpha }_t(x,z;{\mathit{\theta }}_k)\leftarrow {\sum }_{x^{\prime },z^{\prime }}p(x,z|x^{\prime },z^{\prime };{\mathit{\theta }}_k){\alpha }_{t-1}(x^{\prime },z^{\prime };{\mathit{\theta }}_k)$        ${\beta }_t(x,z;{\mathit{\theta }}_k)\leftarrow {\sum }_{x^{\prime },z^{\prime }}{\beta }_{t-1}(x^{\prime },z^{\prime };{\mathit{\theta }}_k)p(x^{\prime },z^{\prime }|x,z;{\mathit{\theta }}_k)$    end for     $F(x,z;{\mathit{\theta }}_k)\leftarrow {\sum }_{t=0}^{T_{max}}{\gamma }^t{\alpha }_t(x,z;{\mathit{\theta }}_k)$     $V(x,z;{\mathit{\theta }}_k)\leftarrow {\sum }_{t=0}^{T_{max}}{\gamma }^t{\beta }_t(x,z;{\mathit{\theta }}_k)$    //—M step—//    Update ${\mathit{\theta }}_k$ to ${\mathit{\theta }}_{k+1}$ by Equations (9)–(11).     $k\leftarrow k+1$ end while return  ${\mathit{\theta }}_k$

4. Bellman EM Algorithm

In the EM algorithm for DEC-POMDP, ${\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)$ and ${\beta }_t(x,\mathit{z};{\mathit{\theta }}_k)$ are calculated from $t=0$ to $t=T_{max}$ to obtain $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$. However, $T_{max}$ needs to be large to reduce the approximation errors of $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$, which impairs the computational efficiency of the EM algorithm for DEC-POMDP [32,33]. In this section, we calculate $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ directly without calculating ${\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)$ and ${\beta }_t(x,\mathit{z};{\mathit{\theta }}_k)$ to resolve the drawback of EM.

4.1. Forward and Backward Bellman Equations

The following equations are useful to obtain $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ directly:

Theorem 2.

$F(x,\mathit{z};\mathit{\theta })$ and $V(x,\mathit{z};\mathit{\theta })$ satisfy the following equations:

$$\begin{array}{cc}\hfill F(x,\mathit{z};\mathit{\theta })=& p_0(x,\mathit{z};\mathit{\nu })+\gamma \sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x,\mathit{z}|x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta })F(x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta }),\hfill \end{array}$$ (29)

$$\begin{array}{cc}\hfill V(x,\mathit{z};\mathit{\theta })=& \overline{r}(x,\mathit{z};\mathit{\pi })+\gamma \sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x^{\prime },z^{\prime }|x,\mathit{z};\mathit{\theta })V(x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta }).\hfill \end{array}$$ (30)

Equations (29) and (30) are called the forward Bellman equation and the backward Bellman equation, respectively.

Proof. 

See Appendix B.1. □

Note that the direction of time is different between Equations (29) and (30). In Equation (29), $x^{\prime }$ and ${\mathit{z}}^{\prime }$ are earlier state and memory than x and z, respectively. In Equation (30), $x^{\prime }$ and ${\mathit{z}}^{\prime }$ are later state and memory than x and z, respectively.

The backward Bellman Equation (30) corresponds to the traditional Bellman equation, which has been used in other algorithms for DEC-POMDP [11,12,13,14]. In contrast, the forward Bellman equation, which is introduced in this paper, is similar to that recently proposed in the offline reinforcement learning [37,38,39].

Since the forward and backward Bellman equations are linear equations, they can be solved exactly as follows:

$$\begin{array}{cc}\hfill \mathit{F}\left(\mathit{\theta }\right)& ={(\mathit{I}-\gamma \mathit{P}\left(\mathit{\theta }\right))}^{-1}\mathit{p}\left(\mathit{\nu }\right),\hfill \end{array}$$ (31)

$$\begin{array}{cc}\hfill \mathit{V}\left(\mathit{\theta }\right)& ={\left({(\mathit{I}-\gamma \mathit{P}\left(\mathit{\theta }\right))}^{-1}\right)}^T\mathit{r}\left(\mathit{\pi }\right)\hfill \end{array}$$ (32)

where

$$\begin{array}{cc}\hfill & F_i\left(\mathit{\theta }\right):=F((x,\mathit{z})=i;\mathit{\theta }),V_i\left(\mathit{\theta }\right):=V((x,\mathit{z})=i;\mathit{\theta })\hfill \\ \hfill & P_{ij}\left(\mathit{\theta }\right):=p((x^{\prime },{\mathit{z}}^{\prime })=i|(x,\mathit{z})=j;\mathit{\theta })\hfill \\ \hfill & p_i\left(\mathit{\nu }\right):=p_0((x,\mathit{z})=i;\mathit{\nu }),r_i\left(\mathit{\pi }\right):=\overline{r}((x,\mathit{z})=i;\mathit{\pi })\hfill \end{array}$$

Therefore, we can obtain $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ from the forward and backward Bellman equations.

4.2. Bellman EM Algorithm (BEM)

The forward–backward algorithm from $t=0$ to $t=T_{max}$ in the EM algorithm for DEC-POMDP can be replaced by the forward and backward Bellman equations. In this paper, the EM algorithm for DEC-POMDP that uses the forward and backward Bellman equations instead of the forward–backward algorithm from $t=0$ to $t=T_{max}$ is called the Bellman EM algorithm (BEM).

4.3. Comparison of EM and BEM

BEM is summarized as Algorithm 2. The M step in Algorithm 2 is almost the same as that in Algorithm 1—only the E step is different. While the time complexity of the E step in EM is ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^2{T}_{max})$, that in BEM is ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^3)$.

Algorithm 2 Bellman EM algorithm (BEM)

$k\leftarrow 0$, Initialize ${\mathit{\theta }}_k$. while ${\mathit{\theta }}_k$ or $J\left({\mathit{\theta }}_k\right)$ do not converge do    Calculate $p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z};{\mathit{\theta }}_k)$ by Equation (20).    //—E step—//     $F\left({\mathit{\theta }}_k\right)\leftarrow {(I-\gamma P\left({\mathit{\theta }}_k\right))}^{-1}p\left({\mathit{\nu }}_k\right)$     $V\left({\mathit{\theta }}_k\right)\leftarrow {\left({(I-\gamma P\left({\mathit{\theta }}_k\right))}^{-1}\right)}^{T}r\left({\mathit{\pi }}_k\right)$    //—M step—//    Update ${\mathit{\theta }}_k$ to ${\mathit{\theta }}_{k+1}$ by Equations (9)–(11).     $k\leftarrow k+1$ end while return  ${\mathit{\theta }}_k$

BEM can calculate $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ exactly. Moreover, BEM can be more efficient than EM when the discount factor $\gamma$ is close to 1 or the acceptable error bound $\epsilon$ is small because $T_{max}$ needs to be large enough in these cases. However, when the size of the state space $\left|\mathcal{X}\right|$ or that of the joint memory space $\left|\mathcal{Z}\right|$ is large, BEM cannot always be more efficient than EM because BEM needs to calculate the inverse matrix ${(\mathit{I}-\gamma \mathit{P}\left({\mathit{\theta }}_k\right))}^{-1}$. To circumvent this shortcoming, we propose a new algorithm, the modified Bellman EM algorithm (MBEM), to obtain $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ without calculating the inverse matrix.

5. Modified Bellman EM Algorithm

5.1. Forward and Backward Bellman Operators

We define the forward and backward Bellman operators as follows:

$$\begin{array}{cc}\hfill A_{\mathit{\theta }}f(x,\mathit{z}):=& p_0(x,\mathit{z};\mathit{\nu })+\gamma \sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x,\mathit{z}|x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta })f(x^{\prime },{\mathit{z}}^{\prime }),\hfill \end{array}$$ (33)

$$\begin{array}{cc}\hfill B_{\mathit{\theta }}v(x,\mathit{z}):=& \overline{r}(x,\mathit{z};\mathit{\pi })+\gamma \sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z};\mathit{\theta })v(x^{\prime },{\mathit{z}}^{\prime })\hfill \end{array}$$ (34)

where ${}^{\forall }f,v:\mathcal{X}\times \mathcal{Z}\to \mathbb{R}$. From the forward and backward Bellman equations, $A_{\mathit{\theta }}$ and $B_{\mathit{\theta }}$ satisfy the following equations:

$$\begin{array}{c}\hfill F(x,\mathit{z};\mathit{\theta })=A_{\mathit{\theta }}F(x,\mathit{z};\mathit{\theta }),\end{array}$$ (35)

$$\begin{array}{c}\hfill V(x,\mathit{z};\mathit{\theta })=B_{\mathit{\theta }}V(x,\mathit{z};\mathit{\theta }).\end{array}$$ (36)

Thus, $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ are the fixed points of $A_{\mathit{\theta }}$ and $B_{\mathit{\theta }}$, respectively. $A_{\mathit{\theta }}$ and $B_{\mathit{\theta }}$ have the following useful property:

Proposition 3.

$A_{\mathit{\theta }}$ and $B_{\mathit{\theta }}$ are contractive operators as follows:

$$\begin{array}{cc}\hfill & \parallel A_{\mathit{\theta }}f-A_{\mathit{\theta }}{g\parallel }_1\le \gamma {\parallel f-g\parallel }_1,\hfill \end{array}$$ (37)

$$\begin{array}{cc}\hfill & \parallel B_{\mathit{\theta }}u-B_{\mathit{\theta }}{v\parallel }_{\infty }\le \gamma {\parallel u-v\parallel }_{\infty }\hfill \end{array}$$ (38)

where ${}^{\forall }f,g,u,v:\mathcal{X}\times \mathcal{Z}\to \mathbb{R}$.

Proof. 

See Appendix C.1. □

Note that the norm is different between Equations (37) and (38). It is caused by the difference of the time direction between $A_{\mathit{\theta }}$ and $B_{\mathit{\theta }}$. While $x^{\prime }$ and ${\mathit{z}}^{\prime }$ are earlier state and memory than x and z, respectively, in the forward Bellman operator $A_{\mathit{\theta }}$, $x^{\prime }$ and ${\mathit{z}}^{\prime }$ are later state and memory than x and z, respectively, in the backward Bellman operator $B_{\mathit{\theta }}$.

We obtain $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ using Equations (35)–(38), as follows:

Proposition 4.

$$\begin{array}{cc}\hfill \underset{L\to \infty }{lim}{A}_{\mathit{\theta }}^{L}f(x,\mathit{z})& =F(x,\mathit{z};\mathit{\theta }),\hfill \end{array}$$ (39)

$$\begin{array}{cc}\hfill \underset{L\to \infty }{lim}{B}_{\mathit{\theta }}^{L}v(x,\mathit{z})& =V(x,\mathit{z};\mathit{\theta })\hfill \end{array}$$ (40)

where ${}^{\forall }f,v:\mathcal{X}\times \mathcal{Z}\to \mathbb{R}$.

Proof. 

See Appendix C.2. □

Therefore, it is shown that $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ can be calculated by applying the forward and backward Bellman operators, $A_{\mathit{\theta }}$ and $B_{\mathit{\theta }}$, to arbitrary initial functions, $f(x,\mathit{z})$ and $v(x,\mathit{z})$, infinite times.

5.2. Modified Bellman EM Algorithm (MBEM)

The calculation of the forward and backward Bellman equations in BEM can be replaced by that of the forward and backward Bellman operators. In this paper, BEM that uses the forward and backward Bellman operators instead of the forward and backward Bellman equations is called modified Bellman EM algorithm (MBEM).

5.3. Comparison of EM, BEM, and MBEM

Since MBEM does not need the inverse matrix, MBEM can be more efficient than BEM when the size of the state space $\left|\mathcal{X}\right|$ and that of the joint memory space $\left|\mathcal{Z}\right|$ are large. Thus, MBEM resolves the drawback of BEM.

On the other hand, MBEM has the same problem as EM. MBEM calculates $A_{{\mathit{\theta }}_k}$ and $B_{{\mathit{\theta }}_k}$ infinite times to obtain $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$. However, since it is practically impossible to calculate $A_{{\mathit{\theta }}_k}$ and $B_{{\mathit{\theta }}_k}$ infinite times, the calculation of $A_{{\mathit{\theta }}_k}$ and $B_{{\mathit{\theta }}_k}$ needs to be terminated after $L_{max}$ times, where $L_{max}$ is finite. In this case, $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ are approximated as follows:

$$\begin{array}{cc}\hfill F(x,\mathit{z};{\mathit{\theta }}_k)=& A_{{\mathit{\theta }}_k}^{\infty }f(x,\mathit{z})\approx A_{{\mathit{\theta }}_k}^{L_{max}}f(x,\mathit{z}),\hfill \end{array}$$ (41)

$$\begin{array}{cc}\hfill V(x,\mathit{z};{\mathit{\theta }}_k)=& B_{{\mathit{\theta }}_k}^{\infty }v(x,\mathit{z})\approx B_{{\mathit{\theta }}_k}^{L_{max}}v(x,\mathit{z}).\hfill \end{array}$$ (42)

$L_{max}$ needs to be large enough to reduce the approximation errors of $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$, which impairs the computational efficiency of MBEM. Thus, MBEM can potentially suffer from the same problem as EM. However, we can theoretically show that MBEM is more efficient than EM by comparing $T_{max}$ and $L_{max}$ under the condition that the approximation errors of $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ are smaller than the acceptable error bound $\epsilon$.

When $f(x,\mathit{z})=p_0(x,\mathit{z};{\mathit{\nu }}_k)$ and $v(x,\mathit{z})=\overline{r}(x,\mathit{z};{\mathit{\pi }}_k)$, Equations (41) and (42) can be calculated as follows:

$$\begin{array}{cc}\hfill A_{{\mathit{\theta }}_k}^{L_{max}}f(x,\mathit{z})& =\sum _{t=0}^{L_{max}}{\gamma }^t{p}_t(x,\mathit{z};\mathit{\theta }),\hfill \end{array}$$ (43)

$$\begin{array}{cc}\hfill B_{{\mathit{\theta }}_k}^{L_{max}}v(x,\mathit{z})& =\sum _{t=0}^{L_{max}}{\gamma }^t{p}_0(o=1|x,\mathit{z},T=t;\mathit{\theta })\hfill \end{array}$$ (44)

which are the same with Equations (23) and (24), respectively. Thus, in this case, $L_{max}=T_{max}$, and the computational efficiency of MBEM is the same as that of EM. However, MBEM has two useful properties that EM does not have, and therefore, MBEM can be more efficient than EM. In the following, we explain these properties in more detail.

The first property of MBEM is the contractibility of the forward and backward Bellman operators, $A_{{\mathit{\theta }}_k}$ and $B_{{\mathit{\theta }}_k}$. From the contractibility of the Bellman operators, $L_{max}$ is determined adaptively as follows:

Proposition 5.

We set an acceptable error bound $\epsilon >0$. If

$$\begin{array}{cc}\hfill & {∥A_{{\mathit{\theta }}_k}^{L}f-A_{{\mathit{\theta }}_k}^{L-1}f∥}_1<\frac{1-\gamma }{\gamma }\epsilon ,\hfill \end{array}$$ (45)

$$\begin{array}{cc}\hfill & {∥B_{{\mathit{\theta }}_k}^{L}v-B_{{\mathit{\theta }}_k}^{L-1}v∥}_{\infty }<\frac{1-\gamma }{\gamma }\epsilon \hfill \end{array}$$ (46)

are satisfied, then

$$\begin{array}{cc}\hfill {∥F(x,z;{\mathit{\theta }}_k)-A_{{\mathit{\theta }}_k}^{L}f(x,z;{\mathit{\theta }}_k)∥}_{\infty }& <\epsilon ,\hfill \end{array}$$ (47)

$$\begin{array}{cc}\hfill {∥V(x,z;{\mathit{\theta }}_k)-B_{{\mathit{\theta }}_k}^{L}v(x,z;{\mathit{\theta }}_k)∥}_{\infty }& <\epsilon \hfill \end{array}$$ (48)

are satisfied.

Proof. 

See Appendix C.3. □

$T_{max}$ is always constant for every E step, $T_{max}=\lceil (log(1-\gamma )\epsilon )/log\gamma -1\rceil$. Thus, even if the approximation errors of $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ are smaller than $\epsilon$ when $t\ll T_{max}$, the forward–backward algorithm cannot be terminated until $t=T_{max}$ because the approximation errors of $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ cannot be evaluated in the forward–backward algorithm.

$L_{max}$ is adaptively determined depending on $A_{{\mathit{\theta }}_k}^{L}f(x,\mathit{z})$ and $B_{{\mathit{\theta }}_k}^{L}v(x,\mathit{z})$. Thus, if $A_{{\mathit{\theta }}_k}^{L}f(x,\mathit{z})$ and $B_{{\mathit{\theta }}_k}^{L}v(x,\mathit{z})$ are close enough to $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$, the E step of MBEM can be terminated because the approximation errors of $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ can be evaluated owing to the contractibility of the forward and backward Bellman operators.

Indeed, when $f(x,\mathit{z})=p_0(x,\mathit{z};{\mathit{\nu }}_k)$ and $v(x,\mathit{z})=\overline{r}(x,\mathit{z};{\mathit{\pi }}_k)$, MBEM is more efficient than EM as follows:

Proposition 6.

When $f(x,\mathit{z})=p_0(x,\mathit{z};{\mathit{\nu }}_k)$ and $v(x,\mathit{z})=\overline{r}(x,\mathit{z};{\mathit{\pi }}_k)$, $L_{max}\le T_{max}$ is satisfied.

Proof. 

See Appendix C.4. □

The second property of MBEM is the arbitrariness of the initial functions, $f(x,\mathit{z})$ and $v(x,\mathit{z})$. In MBEM, the initial functions, $f(x,\mathit{z})$ and $v(x,\mathit{z})$, converge to the fixed points, $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$, by applying the forward and backward Bellman operators, $A_{{\mathit{\theta }}_k}$ and $B_{{\mathit{\theta }}_k}$, $L_{max}$ times. Therefore, if the initial functions, $f(x,\mathit{z})$ and $v(x,\mathit{z})$, are close to the fixed points, $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$, $L_{max}$ can be reduced. Then, the problem is what kind of the initial functions are close to the fixed points.

We suggest that $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$ are set as the initial functions, $f(x,\mathit{z})$ and $v(x,\mathit{z})$. In most cases, ${\mathit{\theta }}_{k-1}$ is close to ${\mathit{\theta }}_k$. When ${\mathit{\theta }}_{k-1}$ is close to ${\mathit{\theta }}_k$, it is expected that $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$ are close to $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$. Therefore, by setting $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$ as the initial functions $f(x,\mathit{z})$ and $v(x,\mathit{z})$, respectively, $L_{max}$ is expected to be reduced. Hence, MBEM can be more efficient than EM because MBEM can utilize the results of the previous iteration, $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$, by this arbitrariness of the initial functions.

However, it is unclear how small $L_{max}$ can be compared to $T_{max}$ by setting $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$ as the initial functions $f(x,\mathit{z})$ and $v(x,\mathit{z})$. Therefore, numerical evaluations are needed. Moreover, in the first iteration, we cannot use the results of the previous iteration, $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$. Therefore, in the first iteration, we set $f(x,\mathit{z})=p_0(x,\mathit{z};{\mathit{\nu }}_k)$ and $v(x,\mathit{z})=\overline{r}(x,\mathit{z};{\mathit{\pi }}_k)$ because these initial functions guarantee $L_{max}\le T_{max}$ from Proposition 6.

MBEM is summarized as Algorithm 3. The M step of Algorithm 3 is exactly the same as that of Algorithms 1 and 2, and only the E step is different. The time complexity of the E step in MBEM is ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^2{L}_{max})$. MBEM does not use the inverse matrix, which resolves the drawback of BEM. Moreover, MBEM can reduce $L_{max}$ by the contractibility of the Bellman operators and the arbitrariness of the initial functions, which can resolve the drawback of EM.

Algorithm 3 Modified Bellman EM algorithm (MBEM)

$k\leftarrow 0$, Initialize ${\mathit{\theta }}_k$. $F(x,z;{\mathit{\theta }}_{k-1})\leftarrow p_0(x,z;{\mathit{\nu }}_k)$ $V(x,z;{\mathit{\theta }}_{k-1})\leftarrow \overline{r}(x,z;{\mathit{\pi }}_k)$ while ${\mathit{\theta }}_k$ or $J\left({\mathit{\theta }}_k\right)$ do not converge do    Calculate $p(x^{\prime },z^{\prime }|x,z;{\mathit{\theta }}_k)$ by Equation (20).    //—E step—//     $F_0(x,z)\leftarrow F(x,z;{\mathit{\theta }}_{k-1})$     $V_0(x,z)\leftarrow F(x,z;{\mathit{\theta }}_{k-1})$     $L\leftarrow 0$    repeat        $F_{L+1}(x,z)\leftarrow A_{{\mathit{\theta }}_k}{F}_L(x,z)$        $V_{L+1}(x,z)\leftarrow B_{{\mathit{\theta }}_k}{V}_L(x,z)$        $L\leftarrow L+1$    until $max\{\parallel F_L-F_{L-1}{\parallel }_1,\parallel V_L-V_{L-1}{\parallel }_{\infty }\}<\frac{1-\gamma }{\gamma }\epsilon$    //—M step—//    Update ${\mathit{\theta }}_k$ to ${\mathit{\theta }}_{k+1}$ by Equations (9)–(11).     $k\leftarrow k+1$ end while return  ${\mathit{\theta }}_k$

6. Summary of EM, BEM, and MBEM

EM, BEM, and MBEM are summarized as in Table 1. The M step is exactly the same among these algorithms, and only the E step is different:

• EM obtains $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ by calculating the forward–backward algorithm up to $T_{max}$. $T_{max}$ needs to be large enough to reduce the approximation errors of $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$, which impairs the computational efficiency.

• BEM obtains $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ by solving the forward and backward Bellman equations. BEM can be more efficient than EM because BEM calculates the forward and backward Bellman equations instead of the forward–backward algorithm up to $T_{max}$. However, BEM cannot always be more efficient than EM when the size of the state $\left|\mathcal{X}\right|$ or that of the memory $\left|\mathcal{Z}\right|$ is large because BEM calculates an inverse matrix to solve the forward and backward Bellman equations.

• MBEM obtains $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ by applying the forward and backward Bellman operators, $A_{{\mathit{\theta }}_k}$ and $B_{{\mathit{\theta }}_k}$, to the initial functions, $f(x,\mathit{z})$ and $v(x,\mathit{z})$, $L_{max}$ times. Since MBEM does not need to calculate the inverse matrix, MBEM may be more efficient than EM even when the size of problems is large, which resolves the drawback of BEM. Although $L_{max}$ needs to be large enough to reduce the approximation errors of $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$, which is the same problem as EM, MBEM can evaluate the approximation errors more tightly owing to the contractibility of $A_{{\mathit{\theta }}_k}$ and $B_{{\mathit{\theta }}_k}$, and can utilize the results of the previous iteration, $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$, as the initial functions, $f(x,\mathit{z})$ and $v(x,\mathit{z})$. These properties enable MBEM to be more efficient than EM.

Table 1.

Summary of EM, BEM, and MBEM.

	EM	Bellman EM (BEM)	Modified Bellman EM (MBEM)
E step	forward–backward algorithm $F(x,\mathit{z};{\mathit{\theta }}_k)\approx {\sum }_{t=0}^{T_{max}}{\gamma }^t{\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)$ $V(x,\mathit{z};{\mathit{\theta }}_k)\approx {\sum }_{t=0}^{T_{max}}{\gamma }^t{\beta }_t(x,\mathit{z};{\mathit{\theta }}_k)$ ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^2{T}_{max})$	forward and backward Bellman equations $\mathit{F}\left({\mathit{\theta }}_k\right)={(\mathit{I}-\gamma \mathit{P}\left({\mathit{\theta }}_k\right))}^{-1}\mathit{p}\left({\mathit{\nu }}_k\right)$ $\mathit{V}\left({\mathit{\theta }}_k\right)={\left({(\mathit{I}-\gamma \mathit{P}\left({\mathit{\theta }}_k\right))}^{-1}\right)}^T\mathit{r}\left({\mathit{\pi }}_k\right)$ ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^3)$	forward and backward Bellman operators $F(x,\mathit{z};{\mathit{\theta }}_k)\approx A_{{\mathit{\theta }}_k}^{L_{max}}f(x,\mathit{z})$ $V(x,\mathit{z};{\mathit{\theta }}_k)\approx B_{{\mathit{\theta }}_k}^{L_{max}}v(x,\mathit{z})$ ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^2{L}_{max})$
M step	Equations (9)–(11) ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^2\left|\mathcal{Y}\right|\left|\mathcal{A}\right|)$	Equations (9)–(11) ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^2\left|\mathcal{Y}\right|\left|\mathcal{A}\right|)$	Equations (9)–(11) ${\mathcal{O}\left(\right(\left|\mathcal{X}\right|\left|\mathcal{Z}\right|)}^2\left|\mathcal{Y}\right|\left|\mathcal{A}\right|)$

7. Numerical Experiment

In this section, we compare the performance of EM, BEM, and MBEM using numerical experiments of four benchmarks for DEC-POMDP: broadcast [40], recycling robot [15], wireless network [27], and box pushing [41]. Detailed settings such as the state transition probability, the observation probability, and the reward function are described at http://masplan.org/problem_domains, accessed on 22 June 2020. We implement EM, BEM, and MBEM in C++.

Figure 3 shows the experimental results. In all the experiments, we set the number of agent $N=2$, the discount factor $\gamma =0.99$, the upper bound of the approximation error $\epsilon =0.1$, and the size of the memory available to the ith agent $|{\mathcal{Z}}^i|=2$. The size of the state $\left|\mathcal{X}\right|$, the action $|{\mathcal{A}}^i|$, and the observation $|{\mathcal{Y}}^i|$ are different for each problem, which are shown on each panel. We note that the size of the state $\left|\mathcal{X}\right|$ is small in the broadcast (a,e,i) and the recycling robot (b,f,j), whereas it is large in the wireless network (c,g,k) and the box pushing (d,h,l).

Figure 3.

Experimental results of four benchmarks for DEC-POMDP: (a,e,i) broadcast; (b,f,j) recycling robot; (c,g,k) wireless network; (d,h,l) box pushing. (a–d) The expected return $J\left({\mathit{\theta }}_k\right)$ as a function of the computational time. (e–h) The expected return $J\left({\mathit{\theta }}_k\right)$ as a function of the iteration k. (i–l) $T_{max}$ and $L_{max}$ as functions of the iteration k. In all the experiments, we set the number of agent $N=2$, the discount factor $\gamma =0.99$, the upper bound of the approximation error $\epsilon =0.1$, and the size of the memory available to the ith agent $|{\mathcal{Z}}^i|=2$. The size of the state $\left|\mathcal{X}\right|$, the action $|{\mathcal{A}}^i|$, and the observation $|{\mathcal{Y}}^i|$ are different for each problem, which are shown on each panel.

While the expected return $J\left({\mathit{\theta }}_k\right)$ with respect to the computational time is different between the algorithms (a–d), that with respect to the iteration k is almost the same (e–h). This is because the M step of these algorithms is exactly the same. Therefore, the difference of the computational time is caused by the computational time of the E step.

The convergence of BEM is faster than that of EM in the small state size problems, i.e., Figure 3a,b. This is because EM calculates the forward–backward algorithm from $t=0$ to $t=T_{max}$, where $T_{max}$ is large. On the other hand, the convergence of BEM is slower than that of EM in the large state size problems, i.e., Figure 3c,d. This is because BEM calculates the inverse matrix.

The convergence of MBEM is faster than that of EM in all the experiments in Figure 3a–d. This is because $L_{max}$ is smaller than $T_{max}$ as shown in Figure 3i–l. While EM requires about 1000 calculations of the forward–backward algorithm to guarantee that the approximation error of $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ is smaller than $\epsilon$, MBEM requires only about 10 calculations of the forward and backward Bellman operators. Thus, MBEM is more efficient than EM. The reason why $L_{max}$ is smaller than $T_{max}$ is that MBEM can utilize the results of the previous iteration, $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$, as the initial functions, $f(x,\mathit{z})$ and $v(x,\mathit{z})$. It is shown from $L_{max}$ and $T_{max}$ in the first iteration. In the first iteration $k=0$, $L_{max}$ is almost the same with $T_{max}$ because $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$ cannot be used as the initial functions $f(x,\mathit{z})$ and $v(x,\mathit{z})$ in the first iteration. On the other hand, in the subsequent iterations $k\ge 1$, $L_{max}$ is much smaller than $T_{max}$ because MBEM can utilize the results of the previous iteration, $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$, the initial functions $f(x,\mathit{z})$ and $v(x,\mathit{z})$.

8. Conclusions and Future Works

In this paper, we propose the Bellman EM algorithm (BEM) and the modified Bellman EM algorithm (MBEM) by introducing the forward and backward Bellman equations into the EM algorithm for DEC-POMDP. BEM can be more efficient than EM because BEM does not calculate the forward–backward algorithm up to the infinite horizon. However, BEM cannot always be more efficient than EM when the size of the state or that of the memory is large because BEM calculates the inverse matrix. MBEM can be more efficient than EM regardless of the size of problems because MBEM does not calculate the inverse matrix. Although MBEM needs to calculate the forward and backward Bellman operators infinite times, MBEM can evaluate the approximation errors more tightly owing to the contractibility of these operators, and can utilize the results of the previous iteration owing to the arbitrariness of initial functions, which enables MBEM to be more efficient than EM. We verified this theoretical evaluation by the numerical experiment, which demonstrates that the convergence of MBEM is much faster than that of EM regardless of the size of problems.

Our algorithms still leave room for further improvements that deal with the real-world problems, which often have a large discrete or continuous state space. Some of them may be addressed by the advanced techniques of the Bellman equations [1,2,3,4]. For example, MBEM may be accelerated by the Gauss–Seidel method [2]. The convergence rate of the E step of MBEM is given by the discount factor $\gamma$, which is the same as that of EM. However, the Gauss–Seidel method modifies the Bellman operators, which allows the convergence rate of MBEM to be smaller than the discount factor $\gamma$. Therefore, even if $F(x,\mathit{z};{\mathit{\theta }}_{k-1})$ and $V(x,\mathit{z};{\mathit{\theta }}_{k-1})$ are not close to $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$, MBEM may be more efficient than EM by the Gauss–Seidel method. Moreover, in DEC-POMDP with a large discrete or continuous state space, $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ cannot be expressed exactly because it requires a large space complexity. This problem may be resolved by the value function approximation [34,35,36]. The value function approximation approximates $F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ using parametric models such as neural networks. The problem is how to find the optimal approximate parameters. The value function approximation finds them by the Bellman equation. Therefore, the potential extensions of our algorithms may lead to the applications to the real-world DEC-POMDP problems.

Appendix A. Proof in Section 3

Appendix A.1. Proof of Theorem 1

$J\left(\mathit{\theta }\right)$ can be calculated as follows:

$$\begin{array}{cc}\hfill J\left(\mathit{\theta }\right):=& {\mathbb{E}}_{\mathit{\theta }}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(x_t,{\mathit{a}}_t)\right]\hfill \\ \hfill =& \sum _{x_{0:\infty },{\mathit{a}}_{0:\infty }}p(x_{0:\infty },{\mathit{a}}_{0:\infty };\mathit{\theta })\sum _{t=0}^{\infty }{\gamma }^{t}r(x_t,{\mathit{a}}_t)=\sum _{t=0}^{\infty }{\gamma }^t\sum _{x_t,{\mathit{a}}_t}p(x_t,{\mathit{a}}_t;\mathit{\theta })r(x_t,{\mathit{a}}_t)\hfill \\ \hfill =& {(1-\gamma )}^{-1}\sum _{T=0}^{\infty }p\left(T\right)\sum _{x_T,{\mathit{a}}_T}p(x_T,{\mathit{a}}_T;\mathit{\theta })\left[(r_{max}-r_{min})p(o=1|x_T,{\mathit{a}}_T)+r_{min}\right]\hfill \\ \hfill =& {(1-\gamma )}^{-1}\left[(r_{max}-r_{min})p(o=1;\mathit{\theta })+r_{min}\right]\hfill \end{array}$$ (A1)

where $x_{0:\infty }:=\{x_0,\dots ,x_{\infty }\}$ and ${\mathit{a}}_{0:\infty }:=\{{\mathit{a}}_0,\dots ,{\mathit{a}}_{\infty }\}$.

Appendix A.2. Proof of Proposition 1

In order to prove Proposition 1, we calculate $Q(\mathit{\theta };{\mathit{\theta }}_k)$. It can be calculated as follows:

$$\begin{array}{cc}\hfill Q(\mathit{\theta };{\mathit{\theta }}_k):=& {\mathbb{E}}_{{\mathit{\theta }}_k}\left[\left.logp(o=1,x_{0:T},y_{0:T},z_{0:T},a_{0:T},T;\mathit{\theta })\right|o=1\right]\hfill \\ \hfill =& {\mathbb{E}}_{{\mathit{\theta }}_k}\left[\left.\sum _{i=1}^N\left\{\sum _{t=0}^{T}log{\pi }^i\left(a_t^i\right|z_t^i)+\sum _{t=1}^{T}log{\lambda }^i\left(z_t^i\right|z_{t-1}^i,y_t^i)+log{\nu }^i\left(z_0^i\right)\right\}\right|o=1\right]+C\hfill \\ \hfill =& \sum _{i=1}^N\left(Q({\pi }^i;{\mathit{\theta }}_k)+Q({\lambda }^i;{\mathit{\theta }}_k)+Q({\nu }^i;{\mathit{\theta }}_k)\right)+C,\hfill \end{array}$$ (A2)

where

$$\begin{array}{cc}\hfill Q({\pi }^i;{\mathit{\theta }}_k)& :={\mathbb{E}}_{{\mathit{\theta }}_k}\left[\left.\sum _{t=0}^{T}log{\pi }^i\left(a_t^i\right|z_t^i)\right|o=1\right],\hfill \end{array}$$ (A3)

$$\begin{array}{cc}\hfill Q({\lambda }^i;{\mathit{\theta }}_k)& :={\mathbb{E}}_{{\mathit{\theta }}_k}\left[\left.\sum _{t=1}^{T}log{\lambda }^i\left(z_t^i\right|z_{t-1}^i,y_t^i)\right|o=1\right],\hfill \end{array}$$ (A4)

$$\begin{array}{cc}\hfill Q({\nu }^i;{\mathit{\theta }}_k)& :={\mathbb{E}}_{{\mathit{\theta }}_k}\left[\left.log{\nu }^i\left(z_0^i\right)\right|o=1\right].\hfill \end{array}$$ (A5)

C is a constant independent of $\mathit{\theta }$.

Proposition A1

([16]). $Q({\pi }^i;{\mathit{\theta }}_k)$, $Q({\lambda }^i;{\mathit{\theta }}_k)$, and $Q({\nu }^i;{\mathit{\theta }}_k)$ are calculated as follows:

$$\begin{array}{cc}\hfill Q({\pi }^i;{\mathit{\theta }}_k)\propto & \sum _{\mathit{a},\mathit{z}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})log{\pi }^i\left(a^i\right|z^i)\sum _{x,x^{\prime },{\mathit{z}}^{\prime }}p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z},\mathit{a};{\mathit{\lambda }}_k)F(x,\mathit{z};{\mathit{\theta }}_k)(\overline{r}(x,\mathit{a})+\gamma V(x^{\prime },{\mathit{z}}^{\prime };{\mathit{\theta }}_k)),\hfill \end{array}$$ (A6)

$$\begin{array}{cc}\hfill Q({\lambda }^i;{\mathit{\theta }}_k)\propto & \sum _{{\mathit{z}}^{\prime },\mathit{z},{\mathit{y}}^{\prime }}{\mathit{\lambda }}_k\left({\mathit{z}}^{\prime }\right|\mathit{z},{\mathit{y}}^{\prime })log{\lambda }^i\left(z^{i^{\prime }}\right|z^i,y^{i^{\prime }})\sum _{x^{\prime },x}p(x^{\prime },{\mathit{y}}^{\prime }|x,\mathit{z};{\mathit{\pi }}_k)F(x,\mathit{z};{\mathit{\theta }}_k)V(x^{\prime },{\mathit{z}}^{\prime };{\mathit{\theta }}_k),\hfill \end{array}$$ (A7)

$$\begin{array}{cc}\hfill Q({\nu }^i;{\mathit{\theta }}_k)\propto & \sum _{\mathit{z}}{p}_0(\mathit{z};{\mathit{\nu }}_n)log{\nu }^i\left(z^i\right)\sum _x{p}_0\left(x\right)V(x,\mathit{z};{\mathit{\theta }}_k).\hfill \end{array}$$ (A8)

$F(x,\mathit{z};{\mathit{\theta }}_k)$ and $V(x,\mathit{z};{\mathit{\theta }}_k)$ are defined by Equations (12) and (13).

Proof. 

Firstly, we prove Equation (A6). $Q({\pi }^i;{\mathit{\theta }}_k)$ can be calculated as follows:

$$\begin{array}{cc}\hfill Q({\pi }^i;{\mathit{\theta }}_k):=& {\mathbb{E}}_{{\mathit{\theta }}_k}\left[\left.\sum _{t=0}^{T}log{\pi }^i\left({\mathit{a}}_t^i\right|{\mathit{z}}_t^i)\right|o=1\right]\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=0}^{\infty }\sum _{x_{0:T},{\mathit{y}}_{0:T},{\mathit{z}}_{0:T},{\mathit{a}}_{0:T}}p(o=1,x_{0:T},{\mathit{y}}_{0:T},{\mathit{z}}_{0:T},{\mathit{a}}_{0:T},T;{\mathit{\theta }}_k)\sum _{t=0}^{T}log{\pi }^i\left({\mathit{a}}_t^i\right|{\mathit{z}}_t^i)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=0}^{\infty }\sum _{t=0}^T\sum _{x_t,{\mathit{z}}_t,{\mathit{a}}_t}p(o=1,x_t,{\mathit{z}}_t,{\mathit{a}}_t,T;{\mathit{\theta }}_k)log{\pi }^i\left({\mathit{a}}_t^i\right|{\mathit{z}}_t^i)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=0}^{\infty }\sum _{t=0}^T\sum _{x_t,{\mathit{z}}_t,{\mathit{a}}_t}p\left(T\right)p_t(x_t,{\mathit{z}}_t;{\mathit{\theta }}_k){\mathit{\pi }}_k\left({\mathit{a}}_t\right|{\mathit{z}}_t)p_t(o=1|x_t,{\mathit{z}}_t,{\mathit{a}}_t,T;{\mathit{\theta }}_k)log{\pi }^i\left({\mathit{a}}_t^i\right|{\mathit{z}}_t^i)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=0}^{\infty }\sum _{t=0}^T\sum _{x,\mathit{z},\mathit{a}}p\left(T\right)p_t(x,\mathit{z};{\mathit{\theta }}_k){\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})p_t(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k)log{\pi }^i\left({\mathit{a}}^i\right|{\mathit{z}}^i)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{x,\mathit{z},\mathit{a}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})log{\pi }^i\left({\mathit{a}}^i\right|{\mathit{z}}^i)\sum _{T=0}^{\infty }p\left(T\right)\sum _{t=0}^T{p}_t(x,\mathit{z};{\mathit{\theta }}_k)p_t(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k).\hfill \end{array}$$ (A9)

Since ${\sum }_{T=0}^{\infty }{\sum }_{t=0}^T\dots ={\sum }_{t=0}^T{\sum }_{T=t}^{\infty }\dots$, we have

$$\begin{array}{cc}\hfill Q({\pi }^i;{\mathit{\theta }}_k)=& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{x,\mathit{z},\mathit{a}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})log{\pi }^i\left(a^i\right|z^i)\sum _{t=0}^{\infty }{p}_t(x,\mathit{z};{\mathit{\theta }}_k)\sum _{T=t}^{\infty }p\left(T\right)p_t(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k)\hfill \\ \hfill =& \frac{1-\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _{x,\mathit{z},\mathit{a}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})log{\pi }^i\left(a^i\right|z^i)\sum _{t=0}^{\infty }{p}_t(x,\mathit{z};{\mathit{\theta }}_k)\sum _{T=t}^{\infty }{\gamma }^T{p}_t(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k).\hfill \end{array}$$ (A10)

Since $p_t(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k)=p_0(o=1|x,\mathit{z},\mathit{a},T-t;$${\mathit{\theta }}_k)$, -4.6cm0cm

$$\begin{array}{cc}\hfill Q({\pi }^i;{\mathit{\theta }}_k)=& \frac{1-\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _{x,\mathit{z},\mathit{a}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})log{\pi }^i\left(a^i\right|z^i)\sum _{t=0}^{\infty }{p}_t(x,\mathit{z};{\mathit{\theta }}_k)\sum _{T=t}^{\infty }{\gamma }^T{p}_0(o=1|x,\mathit{z},\mathit{a},T-t;{\mathit{\theta }}_k)\hfill \\ \hfill =& \frac{1-\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _{x,\mathit{z},\mathit{a}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})log{\pi }^i\left(a^i\right|z^i)\sum _{t=0}^{\infty }{\gamma }^t{p}_t(x,\mathit{z};{\mathit{\theta }}_k)\sum _{T=0}^{\infty }{\gamma }^T{p}_0(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k)\hfill \\ \hfill =& \frac{1-\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _{x,\mathit{z},\mathit{a}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})log{\pi }^i\left(a^i\right|z^i)F(x,\mathit{z};{\mathit{\theta }}_k)\sum _{T=0}^{\infty }{\gamma }^T{p}_0(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k).\hfill \end{array}$$ (A11)

${\sum }_{T=0}^{\infty }{\gamma }^T{p}_0(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k)$ can be calculated as follows:

$$\begin{array}{cc}\hfill \sum _{T=0}^{\infty }{\gamma }^T{p}_0(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k)=& \overline{r}(x,\mathit{a})+\sum _{T=1}^{\infty }{\gamma }^T{p}_0(o=1|x,\mathit{z},\mathit{a},T;{\mathit{\theta }}_k)\hfill \\ \hfill =& \overline{r}(x,\mathit{a})+\sum _{T=1}^{\infty }{\gamma }^T\sum _{x^{\prime },z^{\prime }}{p}_1(o=1|x^{\prime },{\mathit{}}^{\prime },T;{\mathit{\theta }}_k)p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z},\mathit{a};{\mathit{\lambda }}_k)\hfill \\ \hfill =& \overline{r}(x,\mathit{a})+\gamma \sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z},\mathit{a};{\mathit{\lambda }}_k)\sum _{T=0}^{\infty }{\gamma }^T{p}_0(o=1|x^{\prime },{\mathit{z}}^{\prime },T;{\mathit{\theta }}_k)\hfill \\ \hfill =& \overline{r}(x,\mathit{a})+\gamma \sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z},\mathit{a};{\mathit{\lambda }}_k)V(x^{\prime },{\mathit{z}}^{\prime };{\mathit{\theta }}_k).\hfill \end{array}$$ (A12)

From Equations (A11) and (A12), we obtain

$$\begin{array}{cc}\hfill Q({\pi }^i;{\mathit{\theta }}_k)=& \frac{1-\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _{\mathit{a},\mathit{z}}{\mathit{\pi }}_k\left(\mathit{a}\right|\mathit{z})log{\pi }^i\left(a^i\right|z^i)\sum _{x,x^{\prime },{\mathit{z}}^{\prime }}p(x^{\prime },{\mathit{z}}^{\prime }|x,\mathit{z},\mathit{a};{\mathit{\lambda }}_k)F(x,\mathit{z};{\mathit{\theta }}_k)(\overline{r}(x,\mathit{a})+\gamma V(x^{\prime },{\mathit{z}}^{\prime };{\mathit{\theta }}_k)).\hfill \end{array}$$ (A13)

Therefore, Equation (A6) is proved.

Secondly, we prove Equation (A7). $Q({\lambda }^i;{\mathit{\theta }}_k)$ can be calculated as follows:

$$\begin{array}{cc}\hfill Q({\lambda }^i;{\mathit{\theta }}_k):=& {\mathbb{E}}_{{\mathit{\theta }}_k}\left[\left.\sum _{t=1}^{T}log{\lambda }^i\left(z_t^i\right|z_{t-1}^i,y_t^i)\right|o=1\right]\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=1}^{\infty }\sum _{x_{0:T},y_{0:T},z_{0:T},a_{0:T}}p(o=1,x_{0:T},y_{0:T},z_{0:T},a_{0:T},T;{\mathit{\theta }}_k)\sum _{t=1}^{T}log{\lambda }^i\left(z_t^i\right|z_{t-1}^i,y_t^i)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=1}^{\infty }\sum _{t=1}^T\sum _{x_{t-1:t},y_t,z_{t-1:t}}p(o=1,x_{t-1:t},y_t,z_{t-1:t},T;{\mathit{\theta }}_k)log{\lambda }^i\left(z_t^i\right|z_{t-1}^i,y_t^i)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=1}^{\infty }\sum _{t=1}^T\sum _{x_{t-1:t},y_t,z_{t-1:t}}p\left(T\right)p_{t-1}(x_{t-1},z_{t-1};{\mathit{\theta }}_k)p(x_t,y_t|x_{t-1},z_{t-1};{\mathit{\pi }}_k)\hfill \\ \hfill & \times {\mathit{\lambda }}_k\left(z_t\right|z_{t-1},y_t)p_t(o=1|x_t,z_t,T;{\mathit{\theta }}_k)log{\lambda }^i\left(z_t^i\right|z_{t-1}^i,y_t^i)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{x,x^{\prime },y^{\prime },z,z^{\prime }}{\mathit{\lambda }}_k\left(z^{\prime }\right|z,y^{\prime })log{\lambda }^i\left(z^{i^{\prime }}\right|z^i,y^{i^{\prime }})p(x^{\prime },y^{\prime }|x,z;{\mathit{\pi }}_k)\hfill \\ \hfill & \times \sum _{T=1}^{\infty }p\left(T\right)\sum _{t=1}^T{p}_{t-1}(x,z;{\mathit{\theta }}_k)p_t(o=1|x^{\prime },z^{\prime },T;{\mathit{\theta }}_k).\hfill \end{array}$$ (A14)

Since ${\sum }_{T=1}^{\infty }{\sum }_{t=1}^T\dots ={\sum }_{t=1}^{\infty }{\sum }_{T=t}^{\infty }\dots$, we have

$$\begin{array}{cc}\hfill Q({\lambda }^i;{\mathit{\theta }}_k)=& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{x,x^{\prime },y^{\prime },z,z^{\prime }}{\mathit{\lambda }}_k\left(z^{\prime }\right|z,y^{\prime })log{\lambda }^i\left(z^{i^{\prime }}\right|z^i,y^{i^{\prime }})p(x^{\prime },y^{\prime }|x,z;{\mathit{\pi }}_k)\hfill \\ \hfill & \times \sum _{t=1}^{\infty }{p}_{t-1}(x,z;{\mathit{\theta }}_k)\sum _{T=t}^{\infty }p\left(T\right)p_t(o=1|x^{\prime },z^{\prime },T;{\mathit{\theta }}_k)\hfill \\ \hfill =& \frac{1-\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _{x,x^{\prime },y^{\prime },z,z^{\prime }}{\mathit{\lambda }}_k\left(z^{\prime }\right|z,y^{\prime })log{\lambda }^i\left(z^{i^{\prime }}\right|z^i,y^{i^{\prime }})p(x^{\prime },y^{\prime }|x,z;{\mathit{\pi }}_k)\hfill \\ \hfill & \times \sum _{t=1}^{\infty }{p}_{t-1}(x,z;{\mathit{\theta }}_k)\sum _{T=t}^{\infty }{\gamma }^T{p}_t(o=1|x^{\prime },z^{\prime },T;{\mathit{\theta }}_k).\hfill \end{array}$$ (A15)

Since $p_t(o=1,x,z,T;{\mathit{\theta }}_k)=p_0(o=1|x,z,T-t;$${\mathit{\theta }}_k)$,

$$\begin{array}{cc}\hfill Q({\lambda }^i;{\mathit{\theta }}_k)=& \frac{1-\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _{x,x^{\prime },y^{\prime },z,z^{\prime }}{\mathit{\lambda }}_k\left(z^{\prime }\right|z,y^{\prime })log{\lambda }^i\left(z^{i^{\prime }}\right|z^i,y^{i^{\prime }})p(x^{\prime },y^{\prime }|x,z;{\mathit{\pi }}_k)\hfill \\ \hfill & \times \sum _{t=1}^{\infty }{p}_{t-1}(x,z;{\mathit{\theta }}_k)\sum _{T=t}^{\infty }{\gamma }^T{p}_0(o=1|x^{\prime },z^{\prime },T-t;{\mathit{\theta }}_k)\hfill \\ \hfill =& \frac{(1-\gamma )\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _{x,x^{\prime },y^{\prime },z,z^{\prime }}{\mathit{\lambda }}_k\left(z^{\prime }\right|z,y^{\prime })log{\lambda }^i\left(z^{i^{\prime }}\right|z^i,y^{i^{\prime }})p(x^{\prime },y^{\prime }|x,z;{\mathit{\pi }}_k)\hfill \\ \hfill & \times \sum _{t=0}^{\infty }{\gamma }^t{p}_t(x,z;{\mathit{\theta }}_k)\sum _{T=0}^{\infty }{\gamma }^T{p}_0(o=1|x^{\prime },z^{\prime },T;{\mathit{\theta }}_k)\hfill \\ \hfill =& \frac{(1-\gamma )\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _{x,x^{\prime },y^{\prime },z,z^{\prime }}{\mathit{\lambda }}_k\left(z^{\prime }\right|z,y^{\prime })log{\lambda }^i\left(z^{i^{\prime }}\right|z^i,y^{i^{\prime }})p(x^{\prime },y^{\prime }|x,z;{\mathit{\pi }}_k)F(x,z;{\mathit{\theta }}_k)V(x^{\prime },z^{\prime };{\mathit{\theta }}_k).\hfill \end{array}$$ (A16)

Therefore, Equation (A7) is proved.

Finally, we prove Equation (A8). $Q({\nu }^i;{\mathit{\theta }}_k)$ can be calculated as follows:

$$\begin{array}{cc}\hfill Q({\nu }^i;{\mathit{\theta }}_k):=& {\mathbb{E}}_{{\mathit{\theta }}_k}\left[\left.log{\nu }^i\left(z_0^i\right)\right|o=1\right]\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=0}^{\infty }\sum _{x_{0:T},y_{0:T},z_{0:T},a_{0:T}}p(o=1,x_{0:T},y_{0:T},z_{0:T},a_{0:T},T;{\mathit{\theta }}_k)log{\nu }^i\left(z_0^i\right)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=0}^{\infty }\sum _{x_0,z_0}p(o=1,x_0,z_0,T;{\mathit{\theta }}_k)log{\nu }^i\left(z_0^i\right)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=0}^{\infty }\sum _{x_0,z_0}p\left(T\right)p_0\left(x_0\right){\mathit{\nu }}_k\left(z_0\right)p_0(o=1|x_0,z_0,T;{\mathit{\theta }}_k)log{\nu }^i\left(z_0^i\right)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _{T=0}^{\infty }\sum _{x,z}p\left(T\right)p_0\left(x\right){\mathit{\nu }}_k\left(z\right)p_0(o=1|x,z,T;{\mathit{\theta }}_k)log{\nu }^i\left(z^i\right)\hfill \\ \hfill =& \frac{1}{p(o=1;{\mathit{\theta }}_k)}\sum _z{\mathit{\nu }}_k\left(z\right)log{\nu }^i\left(z^i\right)\sum _x{p}_0\left(x\right)\sum _{T=0}^{\infty }p\left(T\right)p_0(o=1|x,z,T;{\mathit{\theta }}_k)\hfill \\ \hfill =& \frac{1-\gamma }{p(o=1;{\mathit{\theta }}_k)}\sum _z{\mathit{\nu }}_k\left(z\right)log{\nu }^i\left(z^i\right)\sum _x{p}_0\left(x\right)V(x,z;{\mathit{\theta }}_k).\hfill \end{array}$$ (A17)

Therefore, Equation (A8) is proved. □

Equations (9)–(11) can be calculated from Equations (A6)–(A8) using the Lagrange multiplier method [16]. Therefore, Proposition 1 is proved.

Appendix A.3. Proof of Proposition 2

The left-hand side of Equation (27) can be calculated as follows:

$$\begin{array}{cc}\hfill & {∥F(x,\mathit{z};{\mathit{\theta }}_k)-\sum _{t=0}^{T_{max}}{\gamma }^t{\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)∥}_{\infty }\hfill \\ \hfill =& {∥\sum _{t=T_{max}+1}^{\infty }{\gamma }^t{\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)∥}_{\infty }\le \sum _{t=T_{max}+1}^{\infty }{\gamma }^t{∥{\alpha }_t(x,\mathit{z};{\mathit{\theta }}_k)∥}_{\infty }=\sum _{t=T_{max}+1}^{\infty }{\gamma }^t=\frac{{\gamma }^{T_{max}+1}}{1-\gamma }.\hfill \end{array}$$ (A18)

Therefore, if

$$\begin{array}{c}\hfill \frac{{\gamma }^{T_{max}+1}}{1-\gamma }<\epsilon \end{array}$$ (A19)

is satisfied, Equation (27) is satisfied. From Equation (A19), we obtain Equation (26). Therefore, Equation (26)⇒Equation (27) is proved. Equation (26)⇒Equation (28) can be proved in the same way.

Appendix B. Proof in Section 4

Proof of Theorem 2

$F(x,\mathit{z};\mathit{\theta })$ can be calculated as follows:

$$\begin{array}{cc}\hfill F(x,\mathit{z};\mathit{\theta }):=& \sum _{t=0}^{\infty }{\gamma }^t{p}_t(x,\mathit{z};\mathit{\theta })\hfill \\ \hfill =& p_0(x,\mathit{z};\mathit{\nu })+\sum _{t=1}^{\infty }{\gamma }^t{p}_t(x,\mathit{z};\mathit{\theta })\hfill \\ \hfill =& p_0(x,\mathit{z};\mathit{\nu })+\sum _{t=1}^{\infty }{\gamma }^t\sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x,\mathit{z}|x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta })p_{t-1}(x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta })\hfill \\ \hfill =& p_0(x,\mathit{z};\mathit{\nu })+\gamma \sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x,\mathit{z}|x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta })\sum _{t=1}^{\infty }{\gamma }^{t-1}{p}_{t-1}(x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta })\hfill \\ \hfill =& p_0(x,\mathit{z};\mathit{\nu })+\gamma \sum _{x^{\prime },{\mathit{z}}^{\prime }}p(x,\mathit{z}|x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta })F(x^{\prime },{\mathit{z}}^{\prime };\mathit{\theta }).\hfill \end{array}$$ (A20)

Therefore, Equation (29) is proved. Equation (30) can be proved in the same way.

Appendix C. Proof in Section 5

Appendix C.1. Proof of Proposition 3

The left-hand side of Equation (37) can be calculated as follows:

$$\begin{array}{cc}\hfill \parallel A_{\mathit{\theta }}f-A_{\mathit{\theta }}{g\parallel }_1& ={\gamma \parallel \mathit{P}\left(\mathit{\theta }\right)\mathit{f}-\mathit{P}\left(\mathit{\theta }\right)\mathit{g}\parallel }_1\hfill \\ \hfill & \le {\gamma \parallel \mathit{P}\left(\mathit{\theta }\right)\parallel }_1{\parallel \mathit{f}-\mathit{g}\parallel }_1\hfill \\ \hfill & ={\gamma \parallel f-g\parallel }_1.\hfill \end{array}$$ (A21)

where $P_{ij}\left(\mathit{\theta }\right):=p((x^{\prime },{\mathit{z}}^{\prime })=i|(x,\mathit{z})=j;\mathit{\theta })$, $f_i:=f((x,\mathit{z})=i)$, and $g_i:=g((x,\mathit{z})=i)$. Equation (38) can be proved in the same way.

Appendix C.2. Proof of Proposition 4

We prove Equation (39) by showing ${lim}_{L\to \infty }{\parallel F\left(\mathit{\theta }\right)-A_{\mathit{\theta }}^{L}f\parallel }_1=0$. Since $F(x,\mathit{z};\mathit{\theta })$ is the fixed point of $A_{\mathit{\theta }}$,

$$\begin{array}{cc}\hfill \parallel F\left(\mathit{\theta }\right)-A_{\mathit{\theta }}^L{f\parallel }_1& =\parallel A_{\mathit{\theta }}^{L}F-A_{\mathit{\theta }}^L{f\parallel }_1.\hfill \end{array}$$ (A22)

From the contractibility of $A_{\mathit{\theta }}$,

$$\begin{array}{c}\hfill \parallel A_{\mathit{\theta }}^{L}F\left(\mathit{\theta }\right)-A_{\mathit{\theta }}^L{f\parallel }_1\le {\gamma }^L{\parallel F\left(\mathit{\theta }\right)-f\parallel }_1.\end{array}$$ (A23)

Since ${\parallel F\left(\mathit{\theta }\right)-f\parallel }_1$ is finite, the right-hand side is 0 when $L\to \infty$. Hence,

$$\begin{array}{cc}\hfill \underset{L\to \infty }{lim}{\parallel F\left(\mathit{\theta }\right)-A_{\mathit{\theta }}^{L}f\parallel }_1& =0\hfill \end{array}$$ (A24)

is satisfied, and Equation (39) is proved. Equation (40) can be proved in the same way.

Appendix C.3. Proof of Proposition 5

From the definition of the norm,

$$\begin{array}{cc}\hfill {∥F\left({\mathit{\theta }}_k\right)-A_{{\mathit{\theta }}_k}^{L}f∥}_{\infty }& \le {∥F\left({\mathit{\theta }}_k\right)-A_{{\mathit{\theta }}_k}^{L}f∥}_1.\hfill \end{array}$$ (A25)

The right-hand side can be calculated as follows:

$$\begin{array}{cc}\hfill {∥F\left({\mathit{\theta }}_k\right)-A_{{\mathit{\theta }}_k}^{L}f∥}_1\le & {∥F\left({\mathit{\theta }}_k\right)-A_{{\mathit{\theta }}_k}^{L+1}f∥}_1+{∥A_{{\mathit{\theta }}_k}^{L+1}f-A_{{\mathit{\theta }}_k}^{L}f∥}_1\hfill \\ \hfill =& {∥A_{{\mathit{\theta }}_k}F\left({\mathit{\theta }}_k\right)-A_{{\mathit{\theta }}_k}^{L+1}f∥}_1+{∥A_{{\mathit{\theta }}_k}^{L+1}f-A_{{\mathit{\theta }}_k}^{L}f∥}_1\hfill \\ \hfill \le & \gamma {∥F\left({\mathit{\theta }}_k\right)-A_{{\mathit{\theta }}_k}^{L}f∥}_1+\gamma {∥A_{{\mathit{\theta }}_k}^{L}f-A_{{\mathit{\theta }}_k}^{L-1}f∥}_1.\hfill \end{array}$$ (A26)

From this inequality, we have

$$\begin{array}{cc}\hfill {∥F\left({\mathit{\theta }}_k\right)-A_{{\mathit{\theta }}_k}^{L}f∥}_1\le & \frac{\gamma }{1-\gamma }{∥A_{{\mathit{\theta }}_k}^{L}f-A_{{\mathit{\theta }}_k}^{L-1}f∥}_1.\hfill \end{array}$$ (A27)

Therefore, if $\parallel A_{{\mathit{\theta }}_k}^{L}f-A_{{\mathit{\theta }}_k}^{L-1}{f\parallel }_1<(1-\gamma ){\gamma }^{-1}\epsilon$ is satisfied,

$$\begin{array}{cc}\hfill {∥F\left({\mathit{\theta }}_k\right)-A_{{\mathit{\theta }}_k}^{L}f∥}_{\infty }& <\epsilon \hfill \end{array}$$ (A28)

holds. Equation (48) is proved in the same way.

Appendix C.4. Proof of Proposition 6

When $f(x,\mathit{z})=p_0(x,\mathit{z};{\mathit{\nu }}_k)$ and $v(x,\mathit{z})=\overline{r}(x,\mathit{z};{\mathit{\pi }}_k)$, the termination conditions of MBEM can be calculated as follows:

$$\begin{array}{cc}\hfill & {∥A_{{\mathit{\theta }}_k}^{L}f-A_{{\mathit{\theta }}_k}^{L-1}f∥}_1\le {\gamma }^{L-1}{∥A_{{\mathit{\theta }}_k}f-f∥}_1={\gamma }^{L-1}{∥\gamma P\left({\mathit{\theta }}_k\right)p\left({\mathit{\nu }}_k\right)∥}_1\le {\gamma }^L{∥P\left({\mathit{\theta }}_k\right)∥}_1{∥p\left({\mathit{\nu }}_k\right)∥}_1={\gamma }^L,\hfill \end{array}$$ (A29)

$$\begin{array}{cc}\hfill & {∥B_{{\mathit{\theta }}_k}^{L}v-B_{{\mathit{\theta }}_k}^{L-1}v∥}_{\infty }\le {\gamma }^L.\hfill \end{array}$$ (A30)

The calculation of $\parallel B_{{\mathit{\theta }}_k}^{L}v-B_{{\mathit{\theta }}_k}^{L-1}{v\parallel }_{\infty }$ is omitted because it is the same as that of $\parallel A_{{\mathit{\theta }}_k}^{L}f-A_{{\mathit{\theta }}_k}^{L-1}{f\parallel }_1$. If

$$\begin{array}{c}\hfill {\gamma }^L<\frac{1-\gamma }{\gamma }\epsilon \left(\Rightarrow L>\frac{log(1-\gamma )\epsilon }{log\gamma }-1\right)\end{array}$$ (A31)

is satisfied,

$$\begin{array}{cc}\hfill & \parallel A_{{\mathit{\theta }}_k}^{L}f-A_{{\mathit{\theta }}_k}^{L-1}{f\parallel }_1<(1-\gamma ){\gamma }^{-1}\epsilon ,\hfill \end{array}$$ (A32)

$$\begin{array}{cc}\hfill & \parallel B_{{\mathit{\theta }}_k}^{L}v-B_{{\mathit{\theta }}_k}^{L-1}{v\parallel }_{\infty }<(1-\gamma ){\gamma }^{-1}\epsilon \hfill \end{array}$$ (A33)

holds. Thus, $L_{max}$ satisfies

$$\begin{array}{c}\hfill L_{max}\le ⌈\frac{log(1-\gamma )\epsilon }{log\gamma }-1⌉.\end{array}$$ (A34)

$\lceil ·\rceil :\mathbb{R}\to \mathbb{Z}$ is defined by $\lceil x\rceil :=min\{n\in \mathbb{Z}|n>x\}$. From Proposition 2, the minimum $T_{max}$ is given by the following equation:

$$\begin{array}{c}\hfill T_{max}=⌈\frac{log(1-\gamma )\epsilon }{log\gamma }-1⌉.\end{array}$$ (A35)

Therefore, $L_{max}\le T_{max}$ holds.

Appendix D. A Note on the Algorithm Proposed by Song et al.

In this section, we show that a parameter dependency is overlooked in the algorithm of [32]. We outline the derivation of the algorithm in [32] and discuss the parameter dependency. Since we use the notation in this paper, it is recommended to read the full paper before reading this section.

Firstly, we calculate the expected return $J\left(\mathit{\theta }\right)$ to derive the algorithm in [32]. Since [32] considers the case where $\mathit{\nu }\left(\mathit{z}\right)={\delta }_{\mathit{z},{\mathit{z}}_0}$, we also consider the same case in this section. The expected return $J\left(\mathit{\theta }\right)$ can be calculated as follows:

$$\begin{array}{cc}\hfill J\left(\mathit{\theta }\right):=& {\mathbb{E}}_{\mathit{\theta }}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(x_t,{\mathit{a}}_t)\right]\hfill \\ \hfill =& \sum _{x_0,{\mathit{a}}_0}p\left(x_0\right)\mathit{\pi }\left({\mathit{a}}_0\right|z_0)\left[r(x_0,{\mathit{a}}_0)+\gamma \sum _{x_1,y_1,z_1}p\left(x_1\right|x_0,{\mathit{a}}_0)p\left(y_1\right|x_1,{\mathit{a}}_0)\mathit{\lambda }\left(z_1\right|z_0,y_1)V(x_1,z_1;\mathit{\theta })\right].\hfill \end{array}$$ (A36)

$V(x,\mathit{z};\mathit{\theta })$ is the value function, which is defined as follows:

$$\begin{array}{c}\hfill V(x,\mathit{z};\mathit{\theta }):={\mathbb{E}}_{\mathit{\theta }}\left[\left.\sum _{t=0}^{\infty }{\gamma }^{t}r(x_t,{\mathit{a}}_t)\right|x_0=x,z_0=z\right].\end{array}$$ (A37)

Equation (A36) can be rewritten as:

$$\begin{array}{cc}\hfill J\left(\mathit{\theta }\right)=& \sum _{a_0^i}{\pi }^i\left(a_0^i\right|z_0^i)\tilde{r}(z_0^i,a_0^i;{\mathit{\pi }}^{-i})+\sum _{a_0^i,y_1^i,z_1^i}{\pi }^i\left(a_0^i\right|z_0^i){\lambda }^i\left(z_1^i\right|z_0^i,y_1^i)\tilde{V}(z_0^i,a_0^i,y_1^i,z_1^i;\mathit{\theta })\hfill \end{array}$$ (A38)

where

$$\begin{array}{cc}\hfill & \tilde{r}(z_0^i,a_0^i;{\mathit{\pi }}^{-i}):=\sum _{x_0,{\mathit{a}}_0^{-i}}p\left(x_0\right){\mathit{\pi }}^{-i}\left({\mathit{a}}_0^{-i}\right|{\mathit{z}}_0^{-i})r(x_0,{\mathit{a}}_0),\hfill \\ \hfill & \tilde{V}(z_0^i,a_0^i,y_1^i,z_1^i;\mathit{\theta })\hfill \end{array}$$ (A39)

$$\begin{array}{cc}\hfill & :=\gamma \sum _{x_0,{\mathit{a}}_0^{-i},x_1,{\mathit{y}}_1^{-i},z_1^{-i}}p\left(x_0\right){\mathit{\pi }}^{-i}\left({\mathit{a}}_0^{-i}\right|{\mathit{z}}_0^{-i})p\left(x_1\right|x_0,{\mathit{a}}_0)p\left({\mathit{y}}_1\right|x_1,{\mathit{a}}_0){\mathit{\lambda }}^{-i}\left({\mathit{z}}_1^{-i}\right|{\mathit{z}}_0^{-i},{\mathit{y}}_1^{-i})V(x_1,{\mathit{z}}_1;\mathit{\theta }).\hfill \end{array}$$ (A40)

Maximizing $J\left(\mathit{\theta }\right)$ is equivalent to maximizing $logJ\left(\mathit{\theta }\right)$, and $logJ\left(\mathit{\theta }\right)$ can be calculated as follows:

$$\begin{array}{cc}\hfill logJ\left(\mathit{\theta }\right)=& log\left\{\sum _{a_0^i}\eta (z_0^i,a_0^i;{\mathit{\theta }}_k)\frac{{\pi }^i\left(a_0^i\right|z_0^i)\tilde{r}(z_0^i,a_0^i;{\mathit{\pi }}^{-i})}{\eta (z_0^i,a_0^i;{\mathit{\theta }}_k)}\right.\hfill \\ \hfill & \left.+\sum _{a_0^i,y_1^i,z_1^i}\rho (z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)\frac{{\pi }^i\left(a_0^i\right|z_0^i){\lambda }^i\left(z_1^i\right|z_0^i,y_1^i)\tilde{V}(z_0^i,a_0^i,y_1^i,z_1^i;\mathit{\theta })}{\rho (z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)}\right\}\hfill \end{array}$$ (A41)

where

$$\begin{array}{cc}\hfill & \eta (z_0^i,a_0^i;{\mathit{\theta }}_k)=\frac{{\pi }_k^i\left(a_0^i\right|z_0^i)\tilde{r}(z_0^i,a_0^i;{\mathit{\pi }}_k^{-i})}{\xi (z_0^i;{\mathit{\theta }}_k)},\hfill \end{array}$$ (A42)

$$\begin{array}{cc}\hfill & \rho (z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)=\frac{{\pi }_k^i\left(a_0^i\right|z_0^i){\lambda }_k^i\left(z_1^i\right|z_0^i,y_1^i)\tilde{V}(z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)}{\xi (z_0^i;{\mathit{\theta }}_k)},\hfill \end{array}$$ (A43)

$$\begin{array}{cc}\hfill & \xi (z_0^i;{\mathit{\theta }}_k)=\sum _{a_0^i}{\pi }_k^i\left(a_0^i\right|z_0^i)\tilde{r}(z_0^i,a_0^i;{\mathit{\pi }}_k^{-i})+\sum _{a_0^i,y_1^i,z_1^i}{\pi }_k^i\left(a_0^i\right|z_0^i){\lambda }_k^i\left(z_1^i\right|z_0^i,y_1^i)\tilde{V}(z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k).\hfill \end{array}$$ (A44)

By the Jensen’s inequality, Equation (A41) can be calculated as follows:

$$\begin{array}{cc}\hfill logJ\left(\mathit{\theta }\right)\ge & \sum _{a_0^i}\eta (z_0^i,a_0^i;{\mathit{\theta }}_k)log\left\{\frac{{\pi }^i\left(a_0^i\right|z_0^i)\tilde{r}(z_0^i,a_0^i;{\mathit{\pi }}^{-i})}{\eta (z_0^i,a_0^i;{\mathit{\theta }}_k)}\right\}\hfill \\ \hfill & +\sum _{a_0^i,y_1^i,z_1^i}\rho (z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)log\left\{\frac{{\pi }^i\left(a_0^i\right|z_0^i){\lambda }^i\left(z_1^i\right|z_0^i,y_1^i)\tilde{V}(z_0^i,a_0^i,y_1^i,z_1^i;\mathit{\theta })}{\rho (z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)}\right\}=:Q(\mathit{\theta };{\mathit{\theta }}_k).\hfill \end{array}$$ (A45)

$logJ\left(\mathit{\theta }\right)=Q(\mathit{\theta };{\mathit{\theta }}_k)$ is satisfied when $\mathit{\theta }={\mathit{\theta }}_k$.

Then, ${\mathit{\theta }}_{k+1}$ is defined as follows:

$$\begin{array}{c}\hfill {\mathit{\theta }}_{k+1}:=arg\underset{\mathit{\theta }}{max}Q(\mathit{\theta };{\mathit{\theta }}_k)\end{array}$$ (A46)

In this case, the following proposition holds:

Proposition A2

([32]). $logJ\left({\mathit{\theta }}_{k+1}\right)\ge logJ\left({\mathit{\theta }}_k\right)$.

Proof. 

From Equation (A45), $logJ\left({\mathit{\theta }}_{k+1}\right)\ge Q({\mathit{\theta }}_{k+1};{\mathit{\theta }}_k)$. From Equation (A46), $Q({\mathit{\theta }}_{k+1};{\mathit{\theta }}_k)\ge Q({\mathit{\theta }}_k;{\mathit{\theta }}_k)=logJ\left({\mathit{\theta }}_k\right)$. Therefore, $logJ\left({\mathit{\theta }}_{k+1}\right)\ge logJ\left({\mathit{\theta }}_k\right)$ is satisfied. □

Therefore, since Equation (A46) monotonically increases $J\left(\mathit{\theta }\right)$, we can find ${\mathit{\theta }}^{\ast }$, which locally maximizes $J\left(\mathit{\theta }\right)$. This is the algorithm in [32].

Then, the problem is how to calculate Equation (A46). It cannot be calculated analytically because $\mathit{\theta }$ dependency of $\tilde{V}(z_0^i,a_0^i,y_1^i,z_1^i;\mathit{\theta })$ is too complex. However, ref. [32] overlooked the parameter dependency of $\tilde{r}(z_0^i,a_0^i;{\mathit{\pi }}^{-i})$ and $\tilde{V}(z_0^i,a_0^i,y_1^i,z_1^i;\mathit{\theta })$, and therefore, it calculated Equation (A46) as follows:

$$\begin{array}{cc}\hfill & {\pi }_{k+1}^i\left(a_0^i\right|z_0^i)=\frac{\eta (z_0^i,a_0^i;{\mathit{\theta }}_k)+{\sum }_{y_1^i,z_1^i}\rho (z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)}{{\sum }_{a_0^i}\left[\eta (z_0^i,a_0^i;{\mathit{\theta }}_k)+{\sum }_{y_1^i,z_1^i}\rho (z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)\right]},\hfill \end{array}$$ (A47)

$$\begin{array}{cc}\hfill & {\lambda }_{k+1}^i\left(z_1^i\right|z_0^i,y_1^i)=\frac{{\sum }_{a_0^i}\rho (z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)}{{\sum }_{a_0^i,z_1^i}\rho (z_0^i,a_0^i,y_1^i,z_1^i;{\mathit{\theta }}_k)}.\hfill \end{array}$$ (A48)

However, Equations (A47) and (A48) do not correspond to Equation (A46), and therefore, the algorithm as a whole may not always provide the optimal policy.

Author Contributions

Conceptualization, T.T., T.J.K.; Formal analysis, T.T., T.J.K.; Funding acquisition, T.J.K.; Writing—original draft, T.T., T.J.K. All authors have read and agreed to the published version of the manuscript.

Funding

This research is supported by JSPS KAKENHI Grant Number 19H05799 and by JST CREST Grant Number JPMJCR2011.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.