Learn Quasi-Stationary Distributions of Finite State Markov Chain

Abstract

We propose a reinforcement learning (RL) approach to compute the expression of quasi-stationary distribution. Based on the fixed-point formulation of quasi-stationary distribution, we minimize the KL-divergence of two Markovian path distributions induced by candidate distribution and true target distribution. To solve this challenging minimization problem by gradient descent, we apply a reinforcement learning technique by introducing the reward and value functions. We derive the corresponding policy gradient theorem and design an actor-critic algorithm to learn the optimal solution and the value function. The numerical examples of finite state Markov chain are tested to demonstrate the new method.

1. Introduction

Quasi-stationary distribution (QSD) is the long time statistical behavior of a stochastic process that will be surely killed when this process is conditioned to survive [1]. This concept has been widely used in applications, such as in biology and ecology [2,3], chemical kinetics [4,5], epidemics [6,7,8], medicine [9] and neuroscience [10,11]. Many works for rare events in meta-stable systems also focus on this quasi-stationary distribution [12,13]. In addition, some new Monte Carlo sampling methods, for instance, the Quasi-stationary Monte Carlo method [14,15], also arise by using QSD instead of true stationary distribution, for instance, the Quasi-stationary Monte Carlo method [14,15]

We are interested in the numerical computation of QSD and focus on the finite state Markov chain in this paper. Mathematically, the quasi-stationary distribution can be solved as the principal left eigenvector of a sub-Markovian transition matrix. Thus, traditional numerical algebra methods can be applied to solve the quasi-stationary distribution in finite state space, for example, the power method [16], the multi-grid method [17] and Arnoldi’s algorithm [18]. These eigenvector methods can produce a stochastic vector for QSD instead of generating samples of QSD.

In search of efficient algorithms for large state space, stochastic approaches are in favor of either sampling the QSD or computing the expression of QSD, and these methods can be applied or extended easily to continuous state space. A popular approach for sampling quasi-stationary distribution is the Fleming–Viot stochastic method [19]. The Flemming–Viot method first simulates N particles independently. When any one of the particles falls into the absorbing state and becomes killed, a new particle is uniformly selected from the remaining $N-1$ surviving particles to replace the dead one, and the simulation continues. When time and N tend to infinity, the particles’ empirical distribution can converge to the quasi-stationary distribution.

In [20,21,22], the authors proposed to recursively update the expression of QSD at each iteration based on the empirical distribution of a single-particle simulation. It is shown in [21] that the convergence rate can be $O\left(n^{-1/2}\right)$, where n is the iteration number. This method is later improved in [23,24] by applying the stochastic approximation method [25] and the Polyak–Ruppert averaging technique [26]. These improved algorithms have a choice of flexible step size but require a projection operator onto probability simplex, which carries some extra computational overhead increasing with the number of states. Ref. [15] extended the algorithm to the diffusion process.

In this paper, we focus on how to compute the expression of the quasi-stationary distribution, which is denoted by $\alpha \left(x\right)$ on a metric space $\mathcal{E}$. If $\mathcal{E}$ is finite, $\alpha$ is a probability vector, and if $\mathcal{E}$ is a domain in ${\mathbb{R}}^d$, then $\alpha$ is a probability density function on $\mathcal{E}$. We assume $\alpha$ can be numerically represented in parametric form ${\alpha }_{\theta }$ and $\theta \in \Theta$. This family $\left\{{\alpha }_{\theta }\right\}$ can be in tabular form or any neural network. Then, the problem of finding the QSD $\alpha$ becomes answering the question of how to compute the optimal parameter $\theta$ in $\Theta$. We call this problem the learning problem for QSD. In addition, we want to directly learn QSD and not use the distribution family $\left\{{\alpha }_{\theta }\right\}$ to fit the simulated samples generated by other traditional simulation methods.

Our minimization problem for QSD is similar to the variational inference (VI) [27], which minimizes an objective functional measuring the distance between the target and candidate distributions. However, unlike the mainstream VI methods such as evidence lower bound (ELBO) technique [28] or particle-based [29], flow-based methods [30], our approach is based on recent important progresses from reinforcement learning (RL) method [31], particularly the policy gradient method and actor-critic algorithm. We first regard the learning process of the quasi-stationary distribution as the interaction with the environment, which is constructed by the property of QSD. Reinforcement learning has recently shown tremendous advancements and remarkable successes in applications (e.g., [32,33,34]). The RL framework provides an innovative and powerful modeling and computation approach for many scientific computing problems.

The essential question is how to formulate the QSD problem as an RL problem. Firstly, for the sub-Markovian kernel K of a Markov process, we can define a Markovian kernel $K_{\alpha }$ on $\mathcal{E}$ (see Definition 1) and then QSD is defined by the equation $\alpha =\alpha K_{\alpha }$, which equals $\alpha$ as the initial distribution and the distribution after one step. Secondly, we consider an optimal $\alpha$ (in our parametric family of distribution) to minimize the Kullback–Leibler divergence (i.e., relative entropy) of two path distributions, denoted by $\mathbb{P}$ and $\mathbb{Q}$, associated with two Markovian kernels $K_{\alpha }$ and $K_{\beta }$ where $\beta :=\alpha K_{\alpha }$. Thirdly, inspired by the recent work [35] of using RL for rare events sampling problems, we transform the minimization of KL divergence between $\mathbb{P}$ and $\mathbb{Q}$ into the maximization of a time-averaged reward function and defined the corresponding value function $V\left(x\right)$ at each state x. This completes our modeling of RL for the quasi-stationary distribution problem. Lastly, we derive the policy gradient theorem (Theorem 1) to compute the gradient with respect to $\theta$ of the averaged reward for the learning dynamic for the averaged reward. This is known as the “actor” part. The “critic” part is to learn the value function V in its parametric form $V_{\psi }$. The actor-critic algorithm uses the stochastic gradient descent to train the parameter $\theta$ for the action ${\alpha }_{\theta }$ and the parameter $\psi$ for the value function $V_{\psi }$ (see Algorithm 1).

Our contribution is that we first devise a method to transform the QSD problem into the RL problem. Similar to [35], our paper also uses the KL-divergence to define the RL problem. However, our paper fully adapts the unique property of QSD that is a fixed point problem $\alpha =\alpha K_{\alpha }$ to define the RL problem.

Our learning method allows the flexible parametrization of the distributions and uses the stochastic gradient method to train the optimal distribution. It is easy to implement optimization with scale up to large state spaces. The numerical examples we tested have shown our that methods converge faster than other existing methods [22,23].

Finally, we remark that our method works very well for QSD of the strict sub-Markovian kernel K but is not applicable to compute the invariant distribution when K is Markovian. This is because we transform the problem into the variational problem between two Markovian kernels $K_{\alpha }$ and $K_{\beta }$ (where $\beta =\alpha K_{\alpha }$). Note that $K_{\alpha }(x,y)=K(x,y)+(1-K(x,\mathcal{E}))\alpha \left(y\right)$ (Definition 1), and our method is based on the fact that $\alpha =\beta$ if and only if $K_{\alpha }=K_{\beta }$. If K is Markovian kernel, then $K_{\alpha }\equiv K$ for any $\alpha$, and our method cannot work. Thus, $K(x,\mathcal{E})$ has to be strictly less than 1 for some $x\in \mathcal{E}$.

This paper is organized as follows. Section 2 is a short review of the quasi-stationary distribution and some basic simulation methods of QSD. In Section 3, we first formulate the reinforcement learning problem by KL-divergence and derive the policy gradient theorem (Theorem 1). Using the above formulation, we then develop the actor-critic algorithm to estimate the quasi-stationary distribution. In Section 4, the efficiency of our algorithms is illustrated by four examples compared with the simulation methods in [24].

Algorithm 1: (ac-$\mathbf{\alpha }$ method) Actor-critic algorithm for quasi-stationary distribution ${\alpha }_{\theta }$

2. Problem Setup and Review

2.1. Quasi-Stationary Distribution

We start with an abstract setting. Let $\mathcal{E}$ be a finite state equipped with the Borel $\sigma$-field $\mathcal{B}\left(\mathcal{E}\right)$, and let $\mathcal{P}\left(\mathcal{E}\right)$ be the space of probabilities over $\mathcal{E}$. A sub-Markovian kernel on $\mathcal{E}$ is defined as a map $K:\mathcal{E}\times \mathcal{B}\left(\mathcal{E}\right)\mapsto [0,1]$ such that for all $x\in \mathcal{E},A\mapsto K(x,A)$ is a nonzero measure with $K(x,\mathcal{E})\le 1$ and for all $A\in \mathcal{B}\left(\mathcal{E}\right),x\mapsto K(x,A)$ is measurable. In particular, if $K(x,\mathcal{E})=1$ for all $x\in \mathcal{E}$, then K is called a Markovian kernel. Throughout the paper, we assume that K is strictly sub-Markovian, i.e., $K(x,\mathcal{E})<1$ for some x.

Let $X_t$ be a Markov chain with values in $\mathcal{E}\cup \left\{\partial \right\}$ where $\partial \notin \mathcal{E}$ denotes an absorbing state. We define the extinction time

$$\tau :=inf\left\{t>0:X_t=\partial \right\}.$$

We define the quasi-stationary distribution (QSD) $\alpha$ as the long time limit of the conditional distribution, if there exists a probability distribution $\nu$ on $\mathcal{E}$ such that the following is the case:

$$\alpha \left(A\right):=\underset{t\to \infty }{lim}{P}_{\nu }\left(X_t\in A\mid \tau >t\right),A\in \mathcal{B}\left(\mathcal{E}\right).$$ (1)

where $P_{\nu }$ refers to the probability distribution of $X_t$ associated with the initial distribution $\nu$ on $\mathcal{E}$. Such a conditional distribution well describes the behavior of the process before extinction, and it is easy to see that $\alpha$ satisfies the following fixed point problem:

$$P_{\alpha }\left(X_t\in A\mid \tau >t\right)=\alpha \left(A\right)$$ (2)

where $P_{\alpha }$ refers to the probability distribution of $X_t$ associated with the initial distribution $\alpha$ on $\mathcal{E}$. Equation (2) is equivalent to the following stationary condition such that the following is the case:

$$\alpha =\frac{\alpha K}{\alpha K\mathbf{1}},\mathrm{or} \alpha \left(y\right)=\frac{{\sum }_x\alpha \left(x\right)K(x,y)}{{\sum }_x\alpha \left(x\right)K(x,\mathcal{E})}$$ (3)

where $\alpha$ is a row vector and $\mathbf{1}$ denotes the column vector with all entries being one and

$$K(x,\mathcal{E})=\sum _{x^{\prime }\in \mathcal{E}}K(x,x^{\prime }).$$

For any sub-Markovian kernel K, we can associate K with a Markovian kernel $\tilde{K}$ on $\mathcal{E}\cup \{\partial \}$ defined by the following:

$$\left\{\begin{array}{c}\tilde{K}(x,A)=K(x,A)\hfill \\ \tilde{K}(x,\{\partial \})=1-K(x,\mathcal{E})\hfill \\ \tilde{K}(\partial ,\{\partial \})=1.\hfill \end{array}\right.$$

for all $x\in \mathcal{E},A\in \mathcal{B}\left(\mathcal{E}\right)$. The kernel $\tilde{K}$ can be understood as the Markovian transition kernel of the Markov chain $\left(X_t\right)$ on $\mathcal{E}\cup \{\partial \}$ for which its transitions in $\mathcal{E}$ is specified by K, but it is “killed” forever once it leaves $\mathcal{E}$.

In this paper, we assume $\mathcal{E}$ is a finite state space and the process in consideration has a unique QSD. Assume that K is irreducible, then existence and uniqueness of the quasi-stationary distribution can be obtained by the Perron–Frobenius theorem [36].

An important Markovian kernel is the following $K_{\alpha }$, which is defined on $\mathcal{E}$ only and has a “regenerative probability” $\alpha$.

Definition 1.

For any given $\alpha \in \mathcal{P}\left(\mathcal{E}\right)$ and a sub-Markovian kernel K on $\mathcal{E}$, we define $K_{\alpha }$, a Markovian kernel on $\mathcal{E}$, as follows:

$$K_{\alpha }(x,A):=K(x,A)+\left(1-K(x,\mathcal{E})\right)\alpha \left(A\right)$$ (4)

for all $x\in \mathcal{E}$ and $A\in \mathcal{B}\left(\mathcal{E}\right)$.

$K_{\alpha }$ is a Markovian kernel because $K_{\alpha }(x,\mathcal{E})=1$. It is easy to sample $X_{t+1}\sim K_{\alpha }(X_t,·)$ from any state $X_t\in \mathcal{E}$: run the transition as normal by using $\tilde{K}$ to have a next state denoted by Y, then $X_{t+1}=Y$ if $Y\in \mathcal{E}$; otherwise, sample $X_{t+1}$ from $\alpha$.

We know that $\alpha$ is the quasi-stationary distribution of K if and only if it is the stationary distribution of $K_{\alpha }$.

$$\alpha =\alpha K_{\alpha }.$$ (5)

It is easy to observe that $\alpha =\beta$ if and only if $K_{\alpha }=K_{\beta }$ for any two distributions $\alpha$ and $\beta$. Moreover, for every ${\alpha }^{\prime }$, $K_{{\alpha }^{\prime }}$ has a unique invariant probability denoted by $\Gamma \left({\alpha }^{\prime }\right)$. Then, ${\alpha }^{\prime }\mapsto \Gamma \left({\alpha }^{\prime }\right)$ is continuous in $\mathcal{P}\left(\mathcal{E}\right)$ (i.e., for the topology of weak convergence), and there exists $\alpha \in \mathcal{P}\left(\mathcal{E}\right)$ such that $\alpha =\Gamma \left(\alpha \right)$ or, equivalently, $\alpha$ is a QSD for K.

2.2. Review of Simulation Methods for Quasi-Stationary Distribution

According to the above subsection, the QSD $\alpha$ satisfies the fixed point problem as follows:

$$\alpha =\Gamma \left(\alpha \right),$$ (6)

where $\Gamma \left(\alpha \right)$ is the stationary distribution of $K_{\alpha }$ on $\mathcal{E}$. In general, (6) can be solved recursively by ${\alpha }_{n+1}\leftarrow \Gamma \left({\alpha }_n\right)$.

The Fleming–Viot (FV) method [19] evolves N particles independently of each other as a Markov process associated with the transition kernel $K_{\alpha }$ until one succeeds in jumping to the absorbing state ∂. At that time, this killed particle is immediately reset to $\mathcal{E}$ as an initial state uniformly chosen from one of the remaining $N-1$ particles. The QSD $\alpha$ is approximated by the empirical distribution of the N particles in total, and these particles can be regarded as samples from the quasi-stationary distribution $\alpha$ such as the MCMC method.

Ref. [37] proposed a simulation method by only using one particle at each iteration to update $\alpha$. At iteration n, given an ${\alpha }_n\in \mathcal{P}\left(\mathcal{E}\right)$, one can run a discrete-time Markov chain $X^{(n+1)}$ as normal on $\partial \cup \mathcal{E}$ with initial $X_0^{(n+1)}\sim {\alpha }_n$; then, ${\alpha }_{n+1}$ is computed as the following weighted average of empirical distributions:

$$\begin{array}{c}\hfill {\alpha }_{n+1}\left(x\right):={\alpha }_n\left(x\right)+\frac{1}{n+1}\sum _{k=0}^{{\tau }^{(n+1)}-1}\frac{I\left(X_k^{(n+1)}=x\mid X_0^{(n+1)}\sim {\alpha }_n\right)-{\alpha }_n\left(x\right)}{\frac{1}{n+1}{\sum }_{j=1}^{n+1}{\tau }^{\left(j\right)}}\end{array}$$ (7)

where $n\ge 0$ and I are the indicator functions, and ${\tau }^{\left(j\right)}=min\left\{k\ge 0\mid X_k^{\left(j\right)}\in \partial \right\}$ is the first extinction time for the process $X^{\left(j\right)}$. This iterative scheme has a convergence rate of $O\left(\frac{1}{\sqrt{n}}\right)$.

In [23,24], the above method is extended to the stochastic approximations framework:

$${\alpha }_{n+1}\left(x\right)={\Theta }_H\left[{\alpha }_n+{ϵ}_n\sum _{k=0}^{{\tau }^{(n+1)}-1}\left(I\left(X_k^{(n+1)}=x|X_0^{(n+1)}\sim {\alpha }_n\right)-{\alpha }_n\left(x\right)\right)\right]$$ (8)

where ${\Theta }_H$ denotes the $L_2$ projection into the probability simplex, and ${ϵ}_n$ is the step size satisfying $\sum {ϵ}_n=\infty$ and $\sum {ϵ}_n^2<\infty$. Specifically, if ${ϵ}_n=O\left(\frac{1}{n^r}\right)$ for $0.5<r<1$, under a sufficient condition, they have $\sqrt{n^r}\left({\alpha }_n-\alpha \right)\stackrel{\mathrm{d}}{\to }\mathcal{N}(0,V)$ for some matrix V [23,24]. If the Polyak–Ruppert averaging technique is applied to generate the following:

$${\nu }_n:=\frac{1}{n}\sum _{k=1}^n{\alpha }_k,$$ (9)

then the convergence rate of ${\nu }_n\to \alpha$ becomes $\frac{1}{\sqrt{n}}$ [23,24].

The simulation schemes (7) and (8) need to sample the initial states according to ${\alpha }_n$ and to add the empirical distribution and ${\alpha }_n$ at each x point wisely. Thus, they are suitable for finite state space where $\alpha$ is a probability vector saved in the tabular form. In (8), there is no need to record all exit times ${\tau }^{\left(j\right)},j=1,\dots ,n$, but the additional projection operation in (8) is computationally expensive since the cost is $O(mlogm)$ where $m=\left|\mathcal{E}\right|$ [38,39].

3. Learn Quasi-Stationary Distribution

We focus on the computation of the expression of the quasi-stationary distribution. In particular, when this distribution is parametrized in a certain manner by $\theta$, we can extend the tabular form for finite-state Markov chain to any flexible form, even in the neural networks for probability density function in ${\mathbb{R}}^d$. However, we do not pursue this representation and expressivity issue here and restrict our discussion to finite state space only to illustrate our main idea first. In finite state space, $\alpha \left(x\right)$ for $x\in \mathcal{E}=\{1,\dots ,m\}$ can be simply described as a softmax function with $m-1$ parameter ${\theta }_i:\alpha \left(i\right)\propto e^{{\theta }_i},1\le i\le m-1$ (${\theta }_m=0$). This introduces no representation error. For the generalization to continuous space $\mathcal{E}$ in jump and diffusion processes or even for a huge finite state space, a good representation of ${\alpha }_{\theta }\left(x\right)$ is important in practice.

In this section, we shall formulate our QSD problem in terms of reinforcement learning (RL) so that the problem of seeking optimal parameters becomes a policy optimization problem. We derive the policy gradient theorem to construct a gradient descent method for the optimal parameter. We then show a method for designing actor-critic algorithms based on stochastic optimization.

3.1. Formulation of RL and Policy Gradient Theorem

Before introducing the RL method of our QSD problem, we develop a general formulation by introducing the KL-divergence between two path distributions.

Let $P_{\theta }$ and $Q_{\theta }$ be two families of Markovian kernels on $\mathcal{E}$ in parametric forms with the same set of parameters $\theta \in \Theta$. Assume both $P_{\theta }$ and $Q_{\theta }$ are ergodic for any $\theta$. Let $T>0$ and denote a path up to time T by ${\omega }_0^T=(X_0,X_1,\dots ,X_T)\in {\mathcal{E}}^{T+1}$. Define the path distributions under the Markov chain kernel $P_{\theta }$ and $Q_{\theta }$, respectively.

$${\mathbb{P}}_{\theta }\left({\omega }_0^T\right):=\prod _{t=1}^T{P}_{\theta }(X_t\mid X_{t-1}),{\mathbb{Q}}_{\theta }\left({\omega }_0^T\right):=\prod _{t=1}^T{Q}_{\theta }(X_t\mid X_{t-1}).$$ (10)

Define the KL divergence from ${\mathbb{P}}_{\theta }$ to ${\mathbb{Q}}_{\theta }$ on ${\mathcal{E}}^{T+1}$:

$${\mathsf{D}}_{KL}({\mathbb{P}}_{\theta }\mid {\mathbb{Q}}_{\theta }):=\sum _{{\omega }_0^T}{\mathbb{P}}_{\theta }\left({\omega }_0^T\right)ln\frac{{\mathbb{P}}_{\theta }\left({\omega }_0^T\right)}{{\mathbb{Q}}_{\theta }\left({\omega }_0^T\right)}=-{\mathbf{E}}_{P_{\theta }}\sum _{t=1}^T{R}_{\theta }(X_{t-1},X_t),$$ (11)

where the expectation ${\mathbf{E}}_{P_{\theta }}$ is for the path $(X_0,X_1,\dots ,X_T)$ generated by the transition kernel $P_{\theta }$, and the following is called the (one-step) reward.

$$R_{\theta }(X_{t-1},X_t):=-ln\frac{P_{\theta }(X_t\mid X_{t-1})}{Q_{\theta }(X_t\mid X_{t-1})}.$$ (12)

Define the average reward $r\left(\theta \right)$ as the time averaged negative KL divergence in the limit of $T\to \infty$.

$$\begin{array}{cc}\hfill r\left(\theta \right)& :=-\underset{T\to \infty }{lim}\frac{1}{T}{\mathsf{D}}_{KL}({\mathbb{P}}_{\theta }\mid {\mathbb{Q}}_{\theta })=-\underset{T\to \infty }{lim}\frac{1}{T}{\mathbf{E}}_{P_{\theta }}\sum _{t=1}^T{R}_{\theta }(X_{t-1},X_t).\hfill \end{array}$$ (13)

Due to ergodicity of $P_{\theta }$, $r\left(\theta \right)={\sum }_{x_0,x_1}{R}_{\theta }(x_0,x_1)P_{\theta }\left(x_1\right|x_0){\mu }_{\theta }\left(x_0\right)$ where ${\mu }_{\theta }$ is the invariant measure of $P_{\theta }$, $r\left(\theta \right)$ is independent of initial state $X_0$. Obviously, $r\left(\theta \right)\le 0$ for any $\theta$.

Property 1.

The following are equivalent:

• 1. $r\left(\theta \right)$ reaches its maximal value 0 at ${\theta }^{*}$;
• 2.  ${\mathbb{P}}_{{\theta }^{*}}={\mathbb{Q}}_{{\theta }^{*}}$ in $\mathcal{P}\left({\mathcal{E}}^{T+1}\right)$ for any $T>0$;
• 3.  $P_{{\theta }^{*}}=Q_{{\theta }^{*}}$;
• 4.  $R_{{\theta }^{*}}\equiv 0$.

Proof. 

We only need to show $\left(1\right)⟹\left(3\right)$. It is easy to see that

$$r\left(\theta \right)=-\sum _{x_0}{\mathsf{D}}_{KL}\left(P_{\theta }(·|x_0)\right|Q_{\theta }(·|x_0)){\mu }_{\theta }\left(x_0\right).$$

If $r\left(\theta \right)=0$, since ${\mu }_{\theta }>0$, then

$${\mathsf{D}}_{KL}\left(P_{\theta }(·|x_0)\right|Q_{\theta }(·|x_0))=0\forall x_0.$$

Thus, we have $P_{\theta }=Q_{\theta }$. □

The above property establishes the relationship between the RL problem and QSD problem.

We show our theoretic main result below as the foundation of our algorithm to be developed later. This theorem can be regarded as one type of the policy gradient theorem for the policy gradient method in reinforcement learning [31].

Define the following value function ([31] Chapter 13).

$$V\left(x\right):=\underset{T\to \infty }{lim}\sum _{t=1}^T{\mathbf{E}}_{P_{\theta }}\left[R_{\theta }(X_{t-1},X_t)-r\left(\theta \right)\mid X_0=x\right].$$ (14)

Certainly, V also depends on $\theta$, although we do not write $\theta$ explicitly.

Theorem 1(policy gradient theorem). 

We have the following two properties:

• 1. 
  At any θ, for any $x\in \mathcal{E}$, the following Bellman-type equation holds for the value function V and the average reward $r\left(\theta \right)$:

  $$V\left(x\right)={\mathbf{E}}_{Y\sim P_{\theta }(·\mid x)}\left[V\left(Y\right)+R_{\theta }(x,Y)-r\left(\theta \right)\right].$$ (15)
• 2. 
  The gradient of the average reward $r\left(\theta \right)$ is the following:

  $$\begin{array}{cc}\hfill {\nabla }_{\theta }r\left(\theta \right)& =\mathbf{E}\left[{\nabla }_{\theta }ln{Q}_{\theta }(Y\mid X)\right]+\hfill \\ \hfill & \mathbf{E}\left[\left(V\left(Y\right)-V\left(X\right)+R_{\theta }(X,Y)-r\left(\theta \right)\right){\nabla }_{\theta }ln{P}_{\theta }(Y\mid X)\right],\hfill \end{array}$$ (16)

where expectations are for the joint distribution $(X,Y)\sim {\mu }_{\theta }\left(x\right)P_{\theta }(y\mid x)$ where ${\mu }_{\theta }$ is the stationary measure of $P_{\theta }$.

Proof. 

We shall prove the Bellman equation first and then we use the Bellman equation to derive the gradient of the average reward $r\left(\theta \right)$. For any $x_0\in \mathcal{E}$, by writing ${\omega }_0^T=(x_0,\dots ,x_T)$ and defining

$$\Delta R_{\theta }\left({\omega }_0^T\right)=\sum _{t=1}^T(R(x_{t-1},x_t)-r\left(\theta \right)),$$

we have the following:

$$\begin{array}{cc}\hfill V\left(x_0\right)& =\underset{T\to \infty }{lim}{\mathbf{E}}_{P_{\theta }}\left[\Delta R_{\theta }\left({\omega }_0^T\right)\mid X_0=x\right]\hfill \\ \hfill & =\underset{T\to \infty }{lim}\sum _{x_2,\dots ,x_T}\sum _{x_1}\left(\left(\prod _{t=2}^T{P}_{\theta }(x_t\mid x_{t-1})\right)P_{\theta }(x_1\mid x_0)\Delta R\left({\omega }_0^T\right)\right)\hfill \\ \hfill & =\underset{T\to \infty }{lim}\sum _{x_1}\left(P_{\theta }(x_1\mid x_0)\sum _{x_2,\dots ,x_T}\left(\prod _{t=2}^T{P}_{\theta }(x_t\mid x_{t-1})\left[\Delta R\left({\omega }_1^T\right)+\Delta R\left({\omega }_0^1\right)\right]\right)\right)\hfill \\ \hfill & =\sum _{x_1}\left(P_{\theta }(x_1\mid x_0)\left(\underset{T\to \infty }{lim}\left[\sum _{x_2,\dots ,x_T}\prod _{t=2}^T{P}_{\theta }(x_t\mid x_{t-1})\Delta R\left({\omega }_1^T\right)\right]+\Delta R\left({\omega }_0^1\right)\right)\right)\hfill \\ \hfill & =\sum _{x_1}{P}_{\theta }(x_1\mid x_0)\left[V\left(x_1\right)+R_{\theta }(x_0,x_1)\right]-r\left(\theta \right),\hfill \end{array}$$ (17)

which proves (15); in other words, we have the following.

$$r\left(\theta \right)={\mathbf{E}}_{Y\sim P_{\theta }(·\mid x)}\left[V\left(Y\right)+R_{\theta }(x,Y)-V\left(x\right)\right],\forall x\in \mathcal{E}.$$

Next, we compute the gradient of $r\left(\theta \right)$. By trivial equality of the following:

$$\sum _{x_1}{P}_{\theta }(x_1\mid x_0){\nabla }_{\theta }ln{P}_{\theta }(x_1\mid x_0)={\nabla }_{\theta }\sum _{x_1}{P}_{\theta }(x_1\mid x_0)=0,$$ (18)

and the definition (12), we can write the gradient of $r\left(\theta \right)$ as follows.

$$\begin{array}{cc}\hfill {\nabla }_{\theta }r\left(\theta \right)=& \sum _y{\nabla }_{\theta }{P}_{\theta }(y\mid x)\left[V\left(y\right)+R_{\theta }(x,y)-V\left(x\right)\right]\hfill \\ \hfill & +\sum _y{P}_{\theta }\left(y\mid x\right)\left[{\nabla }_{\theta }V\left(y\right)-{\nabla }_{\theta }V\left(x\right)+{\nabla }_{\theta }ln{Q}_{\theta }(y\mid x)\right].\hfill \end{array}$$

We here keep the term $V\left(x\right)$ in the first line, even though it has no contribution here (in fact, to add any constant to $V\left(x\right)$ is also fine). Since this equation holds for all states x on the right-hand side, we take the expectation with respect to ${\mu }_{\theta }$, the stationary distribution of $P_{\theta }$. Thus, we have the following.

$$\begin{array}{cc}\hfill {\nabla }_{\theta }r\left(\theta \right)=& \sum _{x,y}{\mu }_{\theta }\left(x\right){\nabla }_{\theta }{P}_{\theta }\left(y\mid x\right)\left[V\left(y\right)+R_{\theta }(x,y)-V\left(x\right)\right]\hfill \\ \hfill & +\sum _{x,y}{\mu }_{\theta }\left(x\right)P_{\theta }\left(y\mid x\right)\left[{\nabla }_{\theta }V\left(y\right)-{\nabla }_{\theta }V\left(x\right)+{\nabla }_{\theta }ln{Q}_{\theta }(y\mid x)\right]\hfill \\ \hfill =& \sum _{x,y}{\mu }_{\theta }\left(x\right){\nabla }_{\theta }{P}_{\theta }\left(y\mid x\right)\left[V\left(y\right)+R_{\theta }(x,y)-V\left(x\right)\right]\hfill \\ \hfill & +\sum _y{\mu }_{\theta }\left(y\right){\nabla }_{\theta }V\left(y\right)-\sum _x{\mu }_{\theta }\left(x\right){\nabla }_{\theta }V\left(x\right)+\sum _{x,y}{\mu }_{\theta }\left(x\right)P_{\theta }\left(y\mid x\right){\nabla }_{\theta }ln{Q}_{\theta }(y\mid x)\hfill \\ \hfill =& \sum _{x,y}{\mu }_{\theta }\left(x\right)P_{\theta }\left(y\mid x\right)\left[V\left(y\right)+R_{\theta }(x,y)-V\left(x\right)\right]{\nabla }_{\theta }ln{P}_{\theta }\left(y\mid x\right)\hfill \\ \hfill & +\sum _{x,y}{\mu }_{\theta }\left(x\right)P_{\theta }\left(y\mid x\right){\nabla }_{\theta }ln{Q}_{\theta }(y\mid x).\hfill \end{array}$$

In fact, we can add any constant number b (independent of x and y) inside the squared bracket of the last line without changing the equality due to the following fact similar to (18): ${\sum }_{x,y}{\mu }_{\theta }\left(x\right){\nabla }_{\theta }{P}_{\theta }\left(y\mid x\right)={\sum }_y{\mu }_{\theta }\left(y\right){\nabla }_{\theta }{\sum }_x{P}_{\theta }\left(x\mid y\right)=0$. (16) is a special case of $b=r\left(\theta \right)$. □

Remark 1.

As shown in the proof, (16) holds if $r\left(\theta \right)$ at the right-hand side is replaced by any constant number b. $b=r\left(\theta \right)$ is a good choice to reduce the variance since $r\left(\theta \right)$ can be regarded as the expectation of $R_{\theta }$.

Remark 2.

If $P_{\theta }=Q_{\theta }$, then the first term of (16) vanishes due to (18) and the second term of (16) vanishes due to (15).

Remark 3.

The name of “policy” here refers to the role of θ as the policy for decision makers to improve reward $r\left(\theta \right)$.

3.2. Learn QSD

Now, we discuss how to connect QSD with the results in the previous subsection. In view of Equation (5), we introduce $\beta :=\alpha K_{\alpha }$ as the one-step distribution if starting from the initial $\alpha$; in other words, we have the following.

$$\beta \left(y\right):=\sum _{x\in \mathcal{E}}\alpha \left(x\right)K_{\alpha }(x,y),\forall y$$ (19)

By (5), $\alpha$ is a QSD if and only if $\beta =\alpha$. However, we do not directly compare these two distributions $\alpha$ and $\beta$. Instead, we consider their Markovian kernels induced by (4): $K_{\alpha }$ and $K_{\beta }$. Our approach is to consider KL divergence similar to (11) between two kernels $K_{\alpha }$ and $K_{\beta }$ since $\alpha =\beta$ if and only if $K_{\alpha }=K_{\beta }$. In this manner, one can view $K_{\alpha }$ and $K_{\beta }$ (note $\beta =\alpha K_{\alpha }$) as two transition matrices $P_{\theta }$ and $Q_{\theta }$ in the previous section, in which the parameter $\theta$ here is in fact the distribution $\alpha$.

To have a further representation of the distribution $\alpha$, which is a (probability mass) function on $\mathcal{E}$, we propose a parametrized family for $\alpha$ in the form ${\alpha }_{\theta }$ where $\theta$ is a generic parameter. In the simplest case, ${\alpha }_{\theta }$ takes the so-called soft-max form ${\alpha }_{\theta }\left(i\right)=\frac{e^{{\theta }_i}}{{\sum }_{j\ge 1}{e}^{{\theta }_j}}$ if $\mathcal{E}=\{1,\dots ,N\}$ for $\theta =({\theta }_1,\dots ,{\theta }_{N-1},{\theta }_N\equiv 0).$ This parametrization represents $\alpha$ without any approximation error for finite state space and the effective space of $\theta$ is just ${\mathbb{R}}^{N-1}$. For certain problems, particularly with large state space, if one has some prior knowledge about the structure of the function $\alpha$ on $\mathcal{E}$, one might propose other parametric forms of ${\alpha }_{\theta }$ with the dimension of $\theta$ less than the cardinality $\left|\mathcal{E}\right|$ to improve the efficiency, although the extra representation error in this manner has to be introduced.

For any given ${\alpha }_{\theta }\in \mathcal{P}\left(\mathcal{E}\right)$, the corresponding Markovian kernel $K_{{\alpha }_{\theta }}$ is then defined in (4) and ${\beta }_{\theta }={\alpha }_{\theta }{K}_{{\alpha }_{\theta }}i$ is defined by (19). $K_{{\beta }_{\theta }}$ is like-wise defined by (4) again. To use the formulation in Section 3.1, we chose $P_{\theta }=K_{{\alpha }_{\theta }}$ and $Q_{\theta }=K_{{\beta }_{\theta }}$. Define the objective function as before:

$$\begin{array}{cc}\hfill r\left(\theta \right)& :=-\underset{T\to \infty }{lim}\frac{1}{T}{\mathsf{D}}_{KL}({\mathbb{P}}_{\theta }\mid {\mathbb{Q}}_{\theta })=-\underset{T\to \infty }{lim}\frac{1}{T}{\mathbf{E}}_{P_{\theta }}\sum _{t=1}^T{R}_{\theta }(X_{t-1},X_t).\hfill \end{array}$$

where the following is the case.

$$R_{\theta }(x,y)=-ln\frac{K_{{\alpha }_{\theta }}(x,y)}{K_{{\beta }_{\theta }}(x,y)}.$$

The value function $V\left(x\right)$ is defined similarly. Theorem 1 now provides the expression of the following gradient:

$$\begin{array}{cc}\hfill {\nabla }_{\theta }r\left(\theta \right)& =\mathbf{E}[\left(R_{\theta }(X,Y)-r\left(\theta \right)+V\left(Y\right)-V\left(X\right)\right){\nabla }_{\theta }ln{K}_{{\alpha }_{\theta }}(X,Y)\hfill \\ & +{\nabla }_{\theta }ln{K}_{{\beta }_{\theta }}(X,Y)]\hfill \end{array}$$ (20)

where $(X,Y)\sim {\mu }_{\theta }\left(x\right)K_{{\alpha }_{\theta }}(x,y)$ and where ${\mu }_{\theta }$ is the stationary measure of $K_{{\alpha }_{\theta }}$.

The optimal ${\theta }^{*}$ for the QSD ${\alpha }_{\theta }$ is to maximize $r\left(\theta \right)$, and this can be solved by the gradient descent algorithm:

$${\theta }_{t+1}={\theta }_t+{\eta }_t^{\theta }{\nabla }_{\theta }r\left({\theta }_t\right).$$ (21)

where ${\eta }_t^{\theta }>0$ is the step size. In practice, the stochastic gradient is applied:

$${\nabla }_{\theta }r\left({\theta }_t\right)\approx {\nabla }_{\theta }ln{K}_{{\alpha }_{\theta }}(X_t,X_{t+1})\times \delta (X_t,X_{t+1})+{\nabla }_{\theta }ln{K}_{{\beta }_{\theta }}(X_t,X_{t+1})$$

where $X_t,X_{t+1}$ are sampled based on the Markovian kernel $K_{{\alpha }_{\theta }}$ (see Algorithm 1) and the differential temporal (TD) error ${\delta }_t$ is as follows.

$${\delta }_t=\delta (X_t,X_{t+1})=R_{\theta }(X_t,X_{t+1})-r\left({\theta }_t\right)+V\left(X_{t+1}\right)-V\left(X_t\right).$$ (22)

Next, we need to address a remaining issue, which is the question of how to compute value functions V and $r\left({\theta }_t\right)$ in the TD error (22). In addition, we also need to show the details of computing ${\nabla }_{\theta }{K}_{{\alpha }_{\theta }}$ and ${\nabla }_{\theta }{K}_{{\beta }_{\theta }}$.

3.3. Actor-Critic Algorithm

With the stochastic gradient method (21), we can obtain optimal policy ${\theta }^{*}$. We refer to (21) as the learning dynamics for the policy, and it is generally known as actor. To calculate the value function V appearing in $\nabla r\left(\theta \right)$, we need to have a new learning dynamic, which is called critic. Then, the overall policy-gradient method is termed as the actor-critic method.

We start with the Bellman Equation (15) for the value function and considered the mean-square-error loss as follows:

$$MSE\left[V\right]=\frac{1}{2}\sum _x\nu \left(x\right){\left(\sum _y{K}_{{\alpha }_{\theta }}(x,y)\left[V\left(y\right)+R_{\theta }(x,y)-r\left(\theta \right)\right]-V\left(x\right)\right)}^2$$

where $\nu$ is any distribution supported on $\mathcal{E}$. $MSE\left[V\right]=0$ if and only if V satisfies the Bellman Equation (15), i.e., V is the value function. To learn V, we introduce function approximation for the value function, $V_{\psi }$, with the parameter $\psi$ and considered to minimize the following:

$$MSE\left(\psi \right)=\frac{1}{2}\sum _x\nu \left(x\right){\left(\sum _y{K}_{{\alpha }_{\theta }}(x,y)\left[V\left(y\right)+R_{\theta }(x,y)-r\left(\theta \right)\right]-V_{\psi }\left(x\right)\right)}^2$$

by the semi-gradient method ([31], Chapter 9).

$$\begin{array}{cc}\hfill {\nabla }_{\psi }MSE\left(\psi \right)& =-\sum _{x,y}\nu \left(x\right)K_{{\alpha }_{\theta }}(x,y)\left[V\left(y\right)+R_{\theta }(x,y)-r\left(\theta \right)-V_{\psi }\left(x\right)\right]{\nabla }_{\psi }{V}_{\psi }\left(x\right)\hfill \\ \hfill & \approx -\sum _{x,y}\nu \left(x\right)K_{{\alpha }_{\theta }}(x,y)\left[V_{\psi }\left(y\right)+R_{\theta }(x,y)-r\left(\theta \right)-V_{\psi }\left(x\right)\right]{\nabla }_{\psi }{V}_{\psi }\left(x\right)\hfill \end{array}$$

Here, the term $V\left(y\right)$ is frozen first and then approximated by $V_{\psi }$ since it could be treated as a prior guess of the value function for the future state.

Then, for the gradient descent iteration ${\psi }_{t+1}={\psi }_t-{\eta }_t^{\psi }{\nabla }_{\psi }{MSE}_V\left({\psi }_t\right)$ where ${\eta }_t^{\psi }$ is the step size, we can have the following stochastic gradient iteration:

$${\psi }_{t+1}={\psi }_t+{\eta }_t^{\psi }\delta (X_t,X_{t+1}){\nabla }_{\psi }{V}_{{\psi }_t}\left(X_t\right)$$ (23)

where the differential temporal (TD) error $\delta$ is defined above in (22).

$${\delta }_t=\delta (X_t,X_{t+1})=R_{{\theta }_t}(X_t,X_{t+1})-r\left({\theta }_t\right)+V_{{\psi }_t}\left(X_{t+1}\right)-V_{{\psi }_t}\left(X_t\right).$$

Here, for the sake of simplicity, $(X_t,X_{t+1})$ are the same samples as in the actor method for ${\theta }_t$. This means that distribution $\nu$ above is chosen as $\mu$ used for the gradient ${\nabla }_{\theta }r\left(\theta \right)$.

Next, we consider the calculation of the reward $r\left(\theta \right)$ by the following Bellman Equation (15).

$$\sum _x\mu \left(x\right)\sum _y{K}_{{\alpha }_{\theta }}(x,y)(R_{\theta }(x,y)-r\left(\theta \right)+V\left(y\right)-V\left(x\right))=0$$

Let $r_t$ be the estimate of the reward $r\left({\theta }_t\right)$ at time t. We can update our estimate of the reward every time a transition occurs as follows:

$$r_{t+1}=r_t+{\eta }_t^r\times {\delta }_t$$ (24)

where ${\delta }_t$ is the TD error before

$${\delta }_t=\delta (X_t,X_{t+1})=R_{{\theta }_t}(X_t,X_{t+1})-r_t+V_{{\psi }_t}\left(X_{t+1}\right)-V_{{\psi }_t}\left(X_t\right).$$

In conclusion, (21), (23) and (24) together consist of the actor-critic algorithm, which is summarized in Algorithm 1. We remark that Algorithm 1 can be easily adapted to use the mini-batch gradient method where several copies of $(X_t,X_{t+1})$ are sampled, and the average is used to update the parameters. The stationary distribution ${\mu }_{\theta }$ of $K_{{\alpha }_{\theta }}$ is sampled by running the corresponding Markov chain for several steps with “warm start”: the initial for ${\theta }_{t+1}$ is set as the final state generated from the previous iteration at ${\theta }_t$. The length of this “burn-in” period can be set as just one step in practice for efficiency.

Remark 4.

Finally, we remark on the computation of ${\nabla }_{\theta }ln{K}_{{\alpha }_{\theta }}$ and ${\nabla }_{\theta }ln{K}_{{\beta }_{\theta }}$ in Algorithm 1. The details are shown in Appendix A. We comment that the main computational cost is the function $K(x,\mathcal{E})$, which has to be pre-computed and stored. If the problem has some special structure, the function could be approximated in parametric form. Another special case is our second example where $K(x,\mathcal{E})=0\forall x\in \{2,3,\dots ,N\}$.

4. Numerical Experiment

In this section, we present two examples to demonstrate Algorithm 1. We call the algorithm (7), (8) and (9) in Section 2.2 used in [23,24], as Vanilla Algorithm, Projection Algorithm and Polyak Averaging Algorithm, respectively. Let 0 be the absorbing state and $\mathcal{E}=\{1,\dots ,N\}$ are non-absorbing states; the Markov transition matrix on $\{0,\dots ,N\}$ is denoted by the following:

$$\tilde{K}=\left[\begin{array}{cc}1& 0\\ *& K\end{array}\right],$$

where K is an N-by-N sub-Markovian matrix. For Algorithm 1, distribution ${\alpha }_{\theta }$ on $\mathcal{E}$ is always parameterized as follows:

$${\alpha }_{\theta }=\frac{1}{e^{{\theta }_1}+\dots +e^{{\theta }_{N-1}}+1}\left[e^{{\theta }_1},\dots ,e^{{\theta }_{N-1}},1\right],$$

and the value function $V_{\psi }\left(x\right)$ is represented in tabular form for simplicity:

$$V_{\psi }=[{\psi }_1,\dots ,{\psi }_N]$$

where $\psi \in {\mathbb{R}}^N$.

4.1. Loopy Markov Chain

We test a toy example of the three-state loopy Markov chain, which was considered in [23,24]. The transition probability matrix for the four states $\{0,1,2,3\}$ is as follows.

$$\tilde{K}=\left[\begin{array}{cccc}1& 0& 0& 0\\ ϵ& \frac{1-ϵ}{3}& \frac{1-ϵ}{3}& \frac{1-ϵ}{3}\\ ϵ& \frac{1-ϵ}{3}& \frac{1-ϵ}{3}& \frac{1-ϵ}{3}\\ ϵ& \frac{1-ϵ}{3}& \frac{1-ϵ}{3}& \frac{1-ϵ}{3}\end{array}\right],ϵ\in (0,1).$$

The state 0 is the absorbing state ∂ and $\mathcal{E}=\{1,2,3\}$. K is the sub-matrix of $\tilde{K}$ corresponding to the states $\{1,2,3\}$. With the probability $ϵ$, the process exits $\mathcal{E}$ directly from state 1, 2 or 3. The true quasi-stationary distribution of this example is the uniform distribution for any $ϵ$.

In order to show the advantage of our algorithm, we consider two cases: (1) $ϵ=0.1$ and (2) $ϵ=0.9$. For a larger $ϵ$, the original Markov chain is very easy to exit; thus, each iteration takes less time, but the convergence rate of Vanilla algorithm is slower.

In order to quantify the accuracy of the learned quasi-stationary distribution, we compute the $L_2$ norm of the error between the learned quasi-stationary distribution and the true values.

In Figure 1, we compute the QSD when $ϵ=0.1$. We set the initial value ${\theta }_0=[-1,1],{\psi }_0=[0,0,0],r_0=0$, the learning rate ${\eta }_n^{\theta }=max\{1/n^{0.1},0.2\},{\eta }_n^{\psi }=0.0001,{\eta }_n^r=0.0001$ and the batch size is 4. The step size for the Projection Algorithm is ${ϵ}_n=n^{-0.99}$. Figure 2 is for the case when $ϵ=0.9$ We set the initial value ${\theta }_0=[4,-2],{\psi }_0=[0,0,0],r_0=0$, the learning rate ${\eta }_n^{\theta }=0.04,{\eta }_n^{\psi }=0.0001,{\eta }_n^r=0.0001$ and the batch size is 32. The step size for the Projection Algorithm is ${ϵ}_n=n^{-0.99}$.

Figure 1.

The loopy Markov chain example with $ϵ=0.1$. The figure shows the log–log plots of $L_2$-norm error of the Vanilla Algorithm (a), Projection Algorithm (b), Polyak Averaging Algorithm (c) and our actor-critic algorithm (d). The iteration for the actor-critic algorithm is defined as one step of gradient descent (“t” in Algorithm 1).

Figure 2.

The loopy Markov chain example with $ϵ=0.9$. The figure shows the log–log plots of $L_2$-norm error of Vanilla Algorithm (a), Projection Algorithm (b), Polyak Averaging Algorithm (c) and our actor-critic algorithm (d).

4.2. M/M/1/N Queue with Finite Capacity and Absorption

Our second example is an M/M/1 queue with finite queue capacity. The 0 state has been set as an absorbing state. The transition probability matrix on $\{0,\dots ,N\}$ takes the following form:

$$\tilde{K}=\left[\begin{array}{cccccccc}1& 0& 0& 0& 0& \dots & 0& 0\\ {\mu }_1& 0& {\lambda }_1& 0& 0& \dots & 0& 0\\ 0& {\mu }_2& 0& {\lambda }_2& 0& \dots & 0& 0\\ 0& 0& {\mu }_3& 0& {\lambda }_3& \dots & 0& 0\\ ⋮& ⋮& ⋮& ⋮& ⋮& & ⋮& ⋮\\ 0& 0& 0& 0& 0& & 0& {\lambda }_{N-1}\\ 0& 0& 0& 0& 0& \dots & 1& 0\end{array}\right]$$

where ${\lambda }_i=\frac{{\rho }_i}{{\rho }_i+1}$, ${\mu }_i=\frac{1}{{\rho }_i+1}$, $i\in \{1,2,\dots ,N-1\}$. ${\rho }_i>1$ means a higher chance to jump to the right than to the left. A larger ${\rho }_i$ will have less probability of exiting $\mathcal{E}$. Note that $K(x,\mathcal{E})=1$ for $x\in \{2,\dots ,N\}$. Thus, $K_{\alpha }(x,y)=K(x,y)$ for any $\alpha$ if $x\ne 1$ and $K_{\alpha }(1,y)=K(1,y)+{\mu }_1\alpha \left(y\right)=\left\{\begin{array}{cc}{\lambda }_1+{\mu }_1\alpha \left(1\right)\hfill & y=1,\hfill \\ {\mu }_1\alpha \left(y\right)\hfill & 2\le y\le N\hfill \end{array}\right..$ Then, $R_{\theta }(x,y)=-ln\frac{K_{{\alpha }_{\theta }}(x,y)}{K_{{\beta }_{\theta }}(x,y)}=0$ if $x\ne 1$ and by (20), the gradient is simplified as follows:

$${\nabla }_{\theta }r\left(\theta \right)={\mathbf{E}}_Y[\left(R_{\theta }(1,Y)-r\left(\theta \right)+V\left(Y\right)-V\left(1\right)\right){\nabla }_{\theta }ln{K}_{{\alpha }_{\theta }}(1,Y)+{\nabla }_{\theta }ln{K}_{{\beta }_{\theta }}(1,Y)]$$

where Y follows distribution $K_{\alpha }(1,·)$.

We consider two cases: (1) a constant ${\rho }_i=1.25$ and (2) a state-dependent ${\rho }_i=2-\frac{3}{2N-4}(i-1)$. Note that ${\rho }_i=1$ gives an equal probability of jumping to the left and to the right. Thus, in case (1), there is a boundary layer at the most right end and in case (2), we expect to see a peak of the QSD near $i\approx 2N/3$. Figure 3 shows the true QSD in both cases. We set $N=500$.

Figure 3.

The QSD for M/M/1/500 queue with ${\rho }_i\equiv 1.25$ (left) and ${\rho }_i=2-\frac{3}{2N-4}(i-1)$ (right).

In Figure 4, we consider the case when ${\rho }_i=1.25$ and compute $L_2$ errors. We set the initial value ${\theta }_0^i=-35+\frac{35}{498}(i-1)$ for $i\in \{1,2,\dots ,498\}$ and ${\theta }_0^{499}=3$, ${\psi }_0=[0,0,\dots ,0]$, $r_0=0$ and the learning rate ${\eta }_n^{\theta }=0.0003,{\eta }_n^{\psi }=0.0001,{\eta }_n^r=0.0001$ and the batch size is 64. The step size for Projection Algorithm is ${ϵ}_n=n^{-0.95}$. Figure 5 plots the errors for the state-dependent ${\rho }_i=2-\frac{3}{2N-4}(i-1)$. We set the initial value ${\theta }_0^i=8+\frac{35}{250}(i-1)$ for $i\in \{1,2,\dots ,250\}$, ${\theta }_0^{251}=44$, ${\theta }_0^i=43$ for $i\in \{252,\dots ,305\}$, ${\theta }_0^{306}=48$, ${\theta }_0^{307}=42$ and ${\theta }_0^i=43-\frac{38}{293}(i-1)$ for $i\in \{308,309,\dots ,499\}$, ${\psi }_0=[0,0,\dots ,0],r_0=0$ and the learning rate is ${\eta }_n^{\theta }=0.0002$, ${\eta }_n^{\psi }=0.0001,{\eta }_n^R=0.0001$ with the batch size as 128. The step size for the Projection Algorithm is ${ϵ}_n=n^{-0.95}$. Both figures demonstrate that actor-critic algorithm performs quite well on this example.

Figure 4.

The M/M/1/500 queue with ${\rho }_i=1.25$. The figure shows the log–log plots of $L_2$-norm error of Vanilla Algorithm (a), Projection Algorithm (b), Polyak Averaging Algorithm (c) and our actor-critic algorithm (d).

Figure 5.

The M/M/1/500 queue with ${\rho }_i=2-\frac{3}{2N-4}(i-1)$. The figure shows the log–log plots of $L_2$-norm error of Vanilla Algorithm (a), Projection Algorithm (b), Polyak Averaging Algorithm (c) and our actor-critic algorithm (d).

In Table 1, we compared the CPU time of each algorithm in the M/M/1/500 queue when they obtain an accuracy at $2\times {10}^{-1}$. We found that our algorithm cost less time on this example.

Table 1.

The CPU time of each algorithm in the M/M/1/500 queue when they obtain the accuracy at $2\times {10}^{-1}$.

Algorithm	Vanilla	Projection	Polyak Averaging	ac_$\mathit{\alpha }$
Time (s)	1038.3279	429.6304	505.2299	186.9280
Time (s)	753.9503	259.0671	268.5476	251.5370

5. Summary and Conclusions

In this paper, we propose a reinforcement learning (RL) method for quasi-stationary distribution (QSD) in discrete time finite-state Markov chains. By minimizing the KL-divergence of two Markovian path distributions induced by the candidate distribution and the true target distribution, we introduce the formulation in terms of RL and derive the corresponding policy gradient theorem. We devise an actor-critic algorithm to learn the QSD in its parameterized form ${\alpha }_{\theta }$. This formulation of RL can receive benefit from the development of the RL method and the optimization theory. We illustrated our actor-critic methods on two numerical examples by using simple tabular parametrization and gradient descent optimization. It has been observed that the performance of our method is more prominent for large scale problems.

We only demonstrate the preliminary mechanism of the idea here, and there is much space left for improving the efficiency and extensions in future works. The generalization from the current consideration of finite-state Markov chain to the jump Markov process and the diffusion case is in consideration. More importantly, for very large or high dimensional state space, modern function approximation methods such as kernel methods or neural networks should be used for the distribution ${\alpha }_{\theta }$ and the value function $V_{\psi }$. The recent tremendous advancement of optimization techniques for policy gradient in reinforcement learning could also contribute much to efficiency improvement of our current formulation.

Appendix A

In this appendix, we discuss the computation of the gradient of ${\nabla }_{\theta }ln{K}_{{\alpha }_{\theta }}$ and ${\nabla }_{\theta }ln{K}_{{\beta }_{\theta }}$. Note that ${\nabla }_{\theta }{\alpha }_{\theta }$ is straightforward since we model $\alpha$ in its parametrization form $\theta$. By definition (4), we have the following:

$${\nabla }_{\theta }ln{K}_{{\alpha }_{\theta }}(X_t,X_{t+1})=\frac{1-K(X_t,\mathcal{E})}{K_{{\alpha }_{\theta }}(X_t,X_{t+1})}{\nabla }_{\theta }{\alpha }_{\theta }\left(X_{t+1}\right)$$

and the following as well:

$${\nabla }_{\theta }ln{K}_{{\beta }_{\theta }}(X_t,X_{t+1})=\frac{1-K(X_t,\mathcal{E})}{K_{{\beta }_{\theta }}(X_t,X_{t+1})}\nabla {\beta }_{\theta }\left(X_{t+1}\right).$$

where $K_{\beta }(x,y)=K(x,y)+(1-K(x,\mathcal{E}))\beta \left(y\right)$. The vector $K(x,\mathcal{E})$ for any x can be pre-computed and saved in tabular form.

By (19), the one-step distribution $\beta$ is computed below.

$$\beta \left(X_{t+1}\right)=\sum _x\alpha \left(x\right)\left[K(x,X_{t+1})+(1-K(x,\mathcal{E}))\alpha \left(X_{t+1}\right)\right]\approx \frac{1}{n}\sum _{i=1}^{n}K(Z_i,X_{t+1})+(1-K(Z_i,\mathcal{E}))\alpha \left(X_{t+1}\right)$$

Here, the samples $Z_i\sim \alpha$ could be approximated by stationary distribution $\mu$; thus, one may simply use the known sample $X_t$ to replace $Z_i$ with $n=1$.

To find ${\nabla }_{\theta }{\beta }_{\theta }$, we use stochastic approximation again.

$$\begin{array}{cc}\hfill {\nabla }_{\theta }{\beta }_{\theta }\left(X_{t+1}\right)& =\sum _x\nabla {\alpha }_{\theta }\left(x\right)\left[K(x,X_{t+1})+(1-K(x,\mathcal{E})){\alpha }_{\theta }\left(X_{t+1}\right)\right]+\left[\sum _x{\alpha }_{\theta }\left(x\right)(1-K(x,\mathcal{E}))\right]{\nabla }_{\theta }{\alpha }_{\theta }\left(y\right),\hfill \\ & \approx \frac{1}{n}\sum _{i=1}^n\nabla ln{\alpha }_{\theta }\left(Z_i\right)\left[K(Z_i,X_{t+1})+(1-K(Z_i,\mathcal{E})){\alpha }_{\theta }\left(X_{t+1}\right)\right]+(1-K(Z_i,\mathcal{E})){\nabla }_{\theta }{\alpha }_{\theta }\left(X_{t+1}\right).\hfill \end{array}$$

Author Contributions

Conceptualization, Z.C. and L.L.; Investigation, Z.C.; Methodology, Z.C. and X.Z.; Software, Z.C.; Supervision, X.Z.; Writing—original draft, Z.C.; Writing—review & editing, L.L. and X.Z. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Government of Hong Kong, Grant Number 11305318; NSFC Grant Number 11871486.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Conflicts of Interest

The authors declare no conflict of interest.