BATCH POLICY LEARNING IN AVERAGE REWARD MARKOV DECISION PROCESSES

Abstract

We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.

1. Introduction.

Mobile health (mHealth) is a rapidly growing field due to the recent advances in mobile and sensing technologies. The mHealth intervention provides a unique opportunity to promote the healthy behaviors (e.g., regular physical activity and adherence to medications) and has been successfully applied in many health fields (e.g., smoking cessation, physical activity, drug abuse and diabetes). Just-in-time adaptive interventions (JITAI, Nahum-Shani et al. (2016)) use a decision rule (i.e., a treatment policy or policy) that maps real-time information about the individual’s context to a particular treatment. In this work we study the problem of how to use data consisting of multiple trajectories to estimate a policy that leads to good long-term performance.

We model the sequential decision making process by a time-homogeneous Markov Decision Process (MDP) (Puterman, 1994) over infinite time horizon. This framework is natural for mobile health applications in which the number of decision times is often large. For example, in HeartSteps, a physical activity mHealth study, there are five decision times per day, resulting in thousands of decision times over a year. Tremendous progress has been made in finite horizon setting; see the recent review by Kosorok and Laber (2019) for references therein. However when the number of time points is very large, methods that are based on the idea of backward iteration (e.g., Q-learning) or importance sampling (Precup, 2000) may suffer a large variance in problems or even be unpractical (Voloshin et al., 2019; Laber et al., 2014).

We propose to estimate the policy that optimizes the long-term average outcomes (rewards) using data consisting of multiple trajectories of finite length. The majority of existing methods focuses on the alternative, the discounted sum of rewards (Sutton and Barto, 2018); see the recent works in statistics (Luckett et al., 2019; Ertefaie and Strawderman, 2018; Shi et al., 2020, 2021). The discounted formulation weighs immediate rewards more heavily than rewards further in the future, which is practical in some applications (e.g., finance). However, for mHealth applications, choosing an appropriate discount rate could be non-trivial. The rewards (i.e., the health outcomes) in the distant future are as important as the near-term ones, especially when considering maintenance of health behaviors as well as longer term treatment burden. This suggests using a large discount rate. However, it is well known that algorithms developed in the discounted setting can become increasingly unstable as the discount rate goes to one; see for example Naik et al. (2019). The long-term average reward framework provides a good approximation to the long-term performance of a desired treatment policy in mHealth. Indeed, it can be shown that under regularity conditions the finite average of the expected rewards converges sublinearly to the long-term average reward as time goes to infinity (Hernández-Lerma and Lasserre, 1999). Therefore, a policy that optimizes the average reward would approximately maximize the sum of the rewards over a sufficiently long time horizon.

In this work, we present a novel algorithm that estimates the optimal policy in a prespecified, parametric policy class. Various methods have been proposed to estimate the global optimal policy by estimating the optimal Q-function;see for example Ormoneit and Sen (2003); Lagoudakis and Parr (2003); Ernst et al. (2005); Munos and Szepesvári (2008); Antos, Szepesvári and Munos (2008a,b); Ertefaie and Strawderman (2018); Fujimoto, Meger and Precup (2019); Kumar et al. (2019); Agarwal, Schuurmans and Norouzi (2020). In practice, the optimal Q-function could be highly non-smooth and complex, thus requiring the use of a flexible function class. This usually results in a learned policy that is also complex. If interpretability is important, this is problematic. Furthermore, when the training data is limited, the flexible function class might overfit the data and thus the variance of the estimated value function and the corresponding policy could be high. Restricting to a pre-specified policy class was studied by Zhang et al. (2012, 2013); Zhou et al. (2017); Zhao et al. (2015, 2019); Athey and Wager (2017) in finite time horizon problems and by Luckett et al. (2019); Murphy et al. (2016); Liu et al. (2019) in infinite time horizon problems. The restriction to a simple policy class enhance the interpretability of the learned policy and reduces the variance of the learned policy, although this induces a bias when the optimal policy is not in the class (i.e., trading off the bias and variance).

To efficiently learn an optimal policy in a prespecified policy class, the main statistical challenge is to construct an estimator for the average reward of a policy that is both data-efficient and performs uniformly well when optimizing over the policy class. Our first contribution of this work is a novel doubly robust estimator (see Section 3); we show that this estimator achieves the semiparametric efficiency bound under certain conditions on the estimation error of nuisance functions (see Section 5). Doubly robust estimators have been developed in the finite time horizon problems (Robins, Rotnitzky and Zhao, 1994; Murphy et al., 2001; Dudík et al., 2014; Jiang and Li, 2016; Thomas and Brunskill, 2016) and recently in the discounted reward infinite horizon setting (Kallus and Uehara, 2019a; Tang et al., 2020). To the best of our knowledge, our doubly robust estimator for the long-term average reward is new. In the literature of the average MDP in the batch setting, only the non-doubly robust estimator proposed by Liao, Klasnja and Murphy (2019) for the long-term average reward can be shown to achieve the semi-parametric efficiency, although they did not explicitly derive it. Most of the previous works on the policy optimization/evaluation under this framework are focused the online setting or under parametric models (e.g., Mahadevan, 1996; Abounadi, Bertsekas and Borkar, 2001; Wan, Naik and Sutton, 2021). Theoretical studies on the average reward MDP in the batch setting are very limited, especially under non-parametric models.

To establish the semiparametric efficiency of the doubly robust estimator and the regret bound, we derive finite-sample error bounds for two nuisance function estimators, a relative value estimator and a ratio estimator. The obtained error bounds are shown to hold uniformly over the prespecified class of policies. Both the relative value and ratio estimators are both derived from the same principle (i.e., coupled estimation; see Section 4). In the case of the ratio estimator, we use an iterative procedure to obtain a near-optimal error bound for the ratio estimator. To the best of our knowledge, this is the first theoretical result characterizing the ratio estimation error, which might be of independent interest.

We learn the optimal policy by maximizing the estimated average reward over a policy class and derive a finite-sample upper bound of the regret. We show that the our proposed method achieves $O(p^{1/2}{n}^{-1/2}+p{n}^{-\tilde{\beta }})$ regret, where p is the number of parameters in the policy, n is the number of trajectories in the training data and $\tilde{\beta }$ is a constant that can be chosen arbitrarily close to 1/(1 + α). Here α ∈ (0, 1) measures the complexity of nuisance function classes. The use of doubly robust estimation ensures the estimation error for the nuisance functions contributes only lower order terms to the regret. Unlike the setting in which the goal is to maximize the average reward defined over a finite horizon (Athey and Wager, 2017), when the goal is to maximize the average reward defined over an infinite horizon, the regret analysis requires uniform control of the estimation error of the policy-dependent nuisance functions over the policy class. We believe this is the first regret bound result for an estimator of an in-class optimal policy in the average reward MDP and using batch data. Recently, Sharma, Jafarnia-Jahromi and Jain (2020) proposed an approximate relative value learning algorithm for globally optimal policies under the average reward MDP with non-parametric function approximation. However, they require the sample size at least exponentially larger than the dimension of the state for the convergence of their algorithm, which seems sub-optimal compared with our result, e.g., Theorem 5.1.

The rest of the article is organized as follows. Section 2 formalizes the decision making problem and introduces the average reward MDP. Section 3 presents the proposed method of learning the in-class optimal policy, including the doubly robust estimator for average reward (Section 3.3). In Section 4, the coupled estimators of the policy-dependent nuisance functions are introduced. Section 5 provides a thorough theoretical analysis on the regret bound of our proposed method. In Section 6, we describe a practical optimization algorithm when Reproducing Kernel Hilbert Spaces (RKHSs) are used to model the nuisance functions. We further conduct several simulation studies to demonstrate the promising performance of our method in Section 7. All the technical proofs are postponed to the supplementary material.

2. Problem Setup.

Suppose we observe a training dataset, ${𝒟}_n={\left\{D_i\right\}}_{i=1}^n$ that consists of n independent, identically distributed (i.i.d.) observations of D:

$$D=\left\{S_1,A_1,S_2,\dots ,S_T,A_T,S_{T+1}\right\}.$$

We use t to index the decision time. The length of the trajectory, T, is a fixed constant. $S_t\in 𝒮$ is the state at time t and $A_t\in 𝒜$ is the action (treatment) selected at time t. We assume the action space, $𝒜$, is finite. To eliminate unnecessary technical distractions, we assume that the state space, $𝒮$, is finite; this assumption imposes no practical limitations and can be extended to the general state space.

The states evolve according to a time-homogeneous Markov process, that is, for t ≥ 1, S_t+1 ⊥ {S₁, A₁, . . . , S_t−1,A_t−1}|{S_t, A_t}, and the conditional distribution does not depend on t. Denote the conditional distribution by P, i.e., Pr(S_t+1 = s′|S_t = s, A_t = a) = P(s′|s, a). The reward (i.e., outcome) is denoted by R_t+1, which is assumed to be a known function of (S_t, A_t, S_t+1), i.e., $R_{t+1}=ℛ\left(S_t,A_t,S_{t+1}\right)$. We assume the reward is bounded, i.e., $\left|ℛ\left(s,a,s^{\prime }\right)\right|\le R_{\text{max}}$. We use r(s,a) to denote the conditional expectation of reward given state and action, i.e., $r(s,a)=\mathbb{E}\left[R_{t+1}\mid S_t=s,A_t=a\right]$.

Let H_t = {S₁, A₁, . . . , S_t} be the history up to time (t − 1) and the current state, S_t. Denote the conditional distribution of A_t given H_t by π_b,t(a|H_t) = Pr(A_t = a|H_t). Let π_b = {π_b,1, . . . , π_b,T}. This is often called behavior policy in the literature. Throughout this paper, the expectation, $\mathbb{E}$, without any subscript is assumed taken with respect to the distribution of the trajectory, D, with the actions selected by the behavior policy π_b.

Consider a time-stationary, Markovian policy, π, that takes the state as input and outputs a probability distribution on the action space, $𝒜$, that is, π(a|s) is the probability of selecting action, a, at state, s. The average reward of the policy, π, is defined as

$${\eta }^{\pi }(s):=\underset{t^{\ast }\to \infty }{\text{lim}}{\mathbb{E}}_{\pi }\left(\frac{1}{t^{\ast }}\sum _{t=1}^{t^{\ast }}{R}_{t+1}\mid S_1=s\right),$$ (2.1)

where the expectation, ${\mathbb{E}}_{\pi }$, is with respect to the distribution of the trajectory in which the states evolve according to P and the actions are chosen by π. In the time-homogeneous MDP with finite state and bounded reward, the limit in (2.1) always exists (Puterman, 1994). The policy, π, induces a Markov chain of states with the transition as $P^{\pi }\left(s^{\prime }|s\right)={\sum }_a\pi (a|s)P\left(s^{\prime }|s,a\right)$. When the induced Markov chain, P^π, is irreducible, it can be shown (e.g., in Puterman (1994)) that the stationary distribution of P^π exists and is unique (denoted by d^π) and the average reward, η^π(s) (2.1) is independent of initial state (denoted by η^π) and equal to

$${\eta }^{\pi }(s)={\eta }^{\pi }=\sum _{s,a}r(s,a)\pi (a|s)d^{\pi }(s).$$ (2.2)

Throughout this paper we consider only the time-stationary, Markovian policies. In fact, it can be shown that the maximal average reward among all possible history dependent policies can be in fact achieved by some time-stationary, Markovian policy (Theorem 8.1.2 in Puterman (1994)). Consider a pre-specified class of such policies, Π, that is parameterized by $\theta \in \text{Θ}\subset {ℝ}^p$. Throughout we assume that the induced Markov chain is always irreducible for any policy in the class, which is summarized below.

Assumption 1.

The induced Markov chain, P^π, is irreducible for π ∈ Π.

The goal of this paper is to develop a method that can efficiently use the training data, ${𝒟}_n$, to learn a policy that maximizes the average reward over Π. We propose to construct ${\widehat{\eta }}_n^{\pi }$, an efficient estimator for the average reward, η^π, for each policy π ∈ Π and learn an optimal policy by solving

$${\widehat{\pi }}_n\in {\text{argmax}}_{\pi \in \text{Π}}{\widehat{\eta }}_n^{\pi }.$$ (2.3)

The performance of ${\widehat{\pi }}_n$ is measured by its regret, defined as

$$\text{Regret}\left({\widehat{\pi }}_n\right)=\underset{\pi \in \text{Π}}{\text{sup}}{\eta }^{\pi }-{\eta }^{{\widehat{\pi }}_n}.$$ (2.4)

Note that although the average reward of the learned policy, ${\widehat{\pi }}_n$, is defined over an infinite horizon, the goal here is to characterize the regret based on using a finite number of trajectories, n, hence the finite sample regret bound is in terms of n. Indeed while the average reward, η^π is defined as t* → ∞ (2.1), the Markovian and stationary assumptions allow us to estimate η^π using fixed length trajectories.

3. Doubly Robust Estimator for Average Reward.

In this section we present a doubly robust estimator for the average reward for a given policy. The estimator is derived from the efficient influence function (EIF). Below we first introduce two functions that occur in the EIF of the average reward. Throughout this section we fix a time-stationary Markovian policy, π, and focus on the setting where the induced Markov chain, P^π, is irreducible (Assumption 1).

3.1. Relative value and ratio functions.

First, we define the relative value function by

$$Q^{\pi }(s,a):=\underset{t^{\ast }\to \infty }{\text{lim}}\frac{1}{t^{\ast }}\sum _{t=1}^{t^{\ast }}{\mathbb{E}}_{\pi }\left[\sum _{k=1}^t\left(R_{k+1}-{\eta }^{\pi }\right)|S_1=s,A_1=a\right].$$ (3.1)

The above limit is well-defined (Puterman (1994), p. 338). If we further assume the induced Markov chain is aperiodic, then the Cesàro limit in (3.1) can be replaced by $Q^{\pi }(s,a)={\mathbb{E}}_{\pi }\left\{{\sum }_{t=1}^{\infty }\left(R_{t+1}-{\eta }^{\pi }\right)|S_1=s,A_1=a\right\}$. Q^π is often called relative value function in that Q^π(s, a) represents the expected total difference between the reward and the average reward under the policy, π, when starting at state, s, and action, a.

The relative value function, Q^π, and the average reward, η^π, are closely related via the Bellman equation:

$${\mathbb{E}}_{\pi }\left[R_{t+1}+Q\left(S_{t+1},A_{t+1}\right)|S_t=s,A_t=a\right]-Q(s,a)-\eta =0.$$ (3.2)

Note that in the above expectation A_t+1 ∼ π(·|S_t+1). It is known that under the irreducibility assumption, the set of solutions of (3.2) is given by $\left\{\left({\eta }^{\pi },Q\right):Q=Q^{\pi }+c\mathbf{1},c\in ℝ\right\}$ where 1(s, a) = 1 for all (s, a); see Puterman (1994), p. 343 for details. As we will see in Section 4.2, the Bellman equation provides the foundation of estimating the relative value function.

We now introduce the ratio function. For t = 1, . . . , T, let d_t(s, a) be the probability mass of state-action pair at time t in the trajectory D generated by the behavior policy. Denote by $d_D(s,a):=(1/T){\sum }_{t=1}^T{d}_t(s,a)$ the average probability mass across the T decision times in D. Similarly, define d_t(s) as the marginal distribution of S_t and $d_D(s):=(1/T){\sum }_{t=1}^T{d}_t(s)$ as the average distribution of states in the trajectory D. Recall that T is the fixed length of the trajectory, D; d_D describes the distribution of this finite length trajectory. Further recall that under Assumption 1, the stationary distribution of P^π exists and is denoted by d^π(s). We assume the following conditions on the data-generating process.

Assumption 2.

The data-generating process satisfies:

• (2–1)There exists some p_min > 0, such that π_b,t(a|H_t) ≥ p_min for all $a\in 𝒜$ and 1 ≤ t ≤ T almost surely.
• (2–2)The average distribution d_D(s) > 0 for all $s\in 𝒮$.

Under Assumption 2, it is easy to see that d_D(s, a) ≥ p_min · (min_s d_D(s)) > 0 for all state-action pair, (s, a). It essentially states that the data generating process ensures that every state-action pair $(s,a)\in 𝒮\times 𝒜$ has a positive probability of being visited, which is a standard assumption in the literature. See e.g., Theorem 7 of Kallus and Uehara (2019b) and (A2) of Shi et al. (2020). In particular, Assumption (2–1) is often satisfied in randomized trials. See our mobile health application in Section 8. In addition, the batch data of our mobile health application consist of 37 trajectories with 210 decision points on each trajectory. In this application, as long as every state has a positive probability of being visited in at least one of 210 decision points, Assumption (2–2) is also satisfied. Assumption (2–2) is imposed on the data generating process. We essentially require that across an infinite number of draws from this data generating process/trajectory, every state $s\in 𝒮$ will be observed. Note that Assumption 2 does not require that the form of the behavior policy is known. Now we can define the ratio function:

$${\omega }^{\pi }(s,a)=\frac{d^{\pi }(s)\pi (a|s)}{d_D(s,a)}$$ (3.3)

The ratio function plays a similar role as the importance weight in finite horizon problems. While the classic importance weight only corrects the distribution of actions between behavior policy and target policy, the ratio here also involves the correction of the states’ distribution. The ratio function is connected with the average reward by

$${\eta }^{\pi }=\mathbb{E}\left\{\frac{1}{T}\sum _{t=1}^T{\omega }^{\pi }\left(S_t,A_t\right)R_{t+1}\right\}$$

for any fixed trajectory length, T. An important property of ω^π is that for any state-action function f(s, a) (not only Q^π),

$$\mathbb{E}\left[\frac{1}{T}\sum _{t=1}^T{\omega }^{\pi }\left(S_t,A_t\right)\left\{f\left(S_t,A_t\right)-\sum _{a^{\prime }}\pi \left(a^{\prime }|S_{t+1}\right)f\left(S_{t+1},a^{\prime }\right)\right\}\right]=0.$$ (3.4)

This orthogonality is key to develop the estimator for ω^π (see Section 4.3).

3.2. Efficient influence function.

In this subsection, we derive the EIF of η^π for a fixed policy π under time-homogeneous Markov Decision Process described in Section 2. Recall that the semiparametric efficiency bound is the supremum of the Cramèr-Rao bounds for all parametric submodels (Newey, 1990). EIF is defined as the influence function of a regular estimator that achieves the semiparametric efficiency bound. For more details, refer to Bickel et al. (1993) and Van der Vaart (2000). The EIF of η^π is given by the following theorem. The proof is provided in Appendix A.

Theorem 3.1.

Suppose the states in the trajectory, D, evolve according to the time-homogeneous Markov process and Assumption 2 holds. Consider a policy, π, such that Assumption 1 holds. Then the EIF of the average reward, η^π, is

$${\varphi }^{\pi }(D)=\frac{1}{T}\sum _{t=1}^T{\omega }^{\pi }\left(S_t,A_t\right)\left\{R_{t+1}+U^{\pi }\left(S_t,A_t,S_{t+1}\right)-{\eta }^{\pi }\right\}$$

where

$$U^{\pi }\left(s,a,s^{\prime }\right):=\sum _{a^{\prime }}\pi \left(a^{\prime }|s^{\prime }\right)Q^{\pi }\left(s^{\prime },a^{\prime }\right)-Q^{\pi }(s,a).$$ (3.5)

Remark 1.

Recall that we impose the Markovian and time-stationary assumptions on the data-generating process. Even though the parameter of interest here (i.e. the average reward η^π) is one-dimensional, there may exist multiple, non-efficient influence functions as a result of these assumptions on the multivariate distribution.

3.3. Doubly robust estimator.

It is known that EIF can be used to derive a semiparametric estimator (see, for example, Chap. 25 in Van der Vaart (2000)). We follow this approach. Specifically, suppose ${\widehat{U}}_n^{\pi }$ and ${\widehat{\omega }}_n^{\pi }$ are estimators of U^π and ω^π respectively. Then we estimate η^π by solving for η in the plug-in estimating equation: ${\mathbb{P}}_n\{(1/T){\sum }_{t=1}^T{\widehat{\omega }}_n^{\pi }(S_t,A_t)\left[R_{t+1}+\right.{\widehat{U}}_n^{\pi }\left(S_t,A_t,S_{t+1}\right)-\eta ]\}=0$, where for any function of the trajectory, f(D), the sample average is denoted as ${\mathbb{P}}_{n}f(D)=(1/n){\sum }_{i=1}^{n}f\left(D_i\right)$. The solution, ${\widehat{\eta }}_n^{\pi }$, is

$${\widehat{\eta }}_n^{\pi }=\frac{{\mathbb{P}}_n\left[(1/T){\sum }_{t=1}^T{\widehat{\omega }}_n^{\pi }\left(S_t,A_t\right)\left\{R_{t+1}+{\widehat{U}}_n^{\pi }\left(S_t,A_t,S_{t+1}\right)\right\}\right]}{{\mathbb{P}}_n\left\{(1/T){\sum }_{t=1}^T{\widehat{\omega }}_n^{\pi }\left(S_t,A_t\right)\right\}}.$$ (3.6)

We have the following doubly robustness of this estimator (the proof is given in Appendix A).

Theorem 3.2.

Suppose ${\widehat{U}}_n^{\pi }(s,a)$ and ${\widehat{\omega }}_n^{\pi }(s,a)$ converge in probability to deterministic limits ${\overline{U}}^{\pi }(s,a)$ and ${\overline{\omega }}^{\pi }(s,a)$ uniformly over $𝒮\times 𝒜$. If either ${\overline{U}}^{\pi }=U^{\pi }$ or ${\overline{\omega }}^{\pi }={\omega }^{\pi }$, then ${\widehat{\eta }}_n^{\pi }$ converges to η^π in probability.

Remark 2.

The uniform convergence in probability can be relaxed to L₂ convergence by using uniform laws of large numbers. The doubly robustness can protect against potential model mis-specifications since we only require one of two models is correct. Moreover, the doubly robust structure can be used to relax the required rate for each of the nuisance function estimation to achieve the semiparametric efficiency bound, especially if we use sample-splitting techniques (see Section 9), as discussed in Chernozhukov et al. (2018).

4. Estimators for the Nuisance Functions.

Recall the doubly robust estimator (3.6) requires the estimation of two nuisance functions, U^π and ω^π. It turns out that although these two nuisance functions are defined from different perspectives, both nuisance functions can in fact be characterized in a similar way. Both estimators can be obtained by minimizing an objective function that involves a minimizer of another objective function (“coupled estimation”). This can be viewed as a generalization of the classical M-estimator with a “plug-in estimator” in the sense that the the second objective function also involves the unknown parameters to be estimated. The idea of coupled estimation was previously used by Antos, Szepesvári and Munos (2008a); Farahmand et al. (2016) to estimate the value function in the discounted reward setting and recently by Liao, Klasnja and Murphy (2019) in the average reward setting. In what follows we provide a general coupled estimation framework and discuss the motivation for using it. We then review the coupled estimator for relative value function and ratio function in Liao, Klasnja and Murphy (2019).

4.1. Review of coupled estimation.

Consider a setting where the true parameter (or function), θ*, can be characterized as the minimizer of the following objective function:

$${\theta }^{\ast }={\text{argmin}}_{\theta }J(\theta )=\mathbb{E}\left\{\left(l_1\circ f_{\theta }\right)(Z)\right\}$$ (4.1)

where $l_1:ℝ\to {ℝ}^{+}$ is a loss function composite with f (e.g., the squared loss, l₁(x) = x² and the linear model, f_θ(Z) = Y −θ^⊤X, where Z = (X, Y)). If we can directly evaluate f_θ(Z), then we can estimate θ* by the classical M-estimator, ${\text{argmin}}_{\theta }{\mathbb{P}}_n\left\{\left(l_1\circ f_{\theta }\right)(Z)\right\}$.

In our setting f_θ is of the form $f_{\theta }(Z)=\mathbb{E}\left[F_{\theta }\left(Z^{\prime }\right)|Z\right]$ and f_θ(Z) cannot be directly evaluated because we don’t have an explicit formula for the conditional expectation $\mathbb{E}\left[F_{\theta }\left(Z^{\prime }\right)|Z\right]$. A natural idea to remedy this is to replace the unknown f_θ(Z) by F_θ(Z′) and estimate θ* by ${\text{argmin}}_{\theta }{\mathbb{P}}_n\left\{\text{(}{l}_1\circ F_{\theta }\left(Z^{\prime }\right)\right\}$. Unfortunately this estimator is biased in general. To see this, suppose l₁(x) = x². We note that the limit of the new objective function, ${\mathbb{P}}_n\left\{\text{(}{l}_1\circ F_{\theta }\left(Z^{\prime }\right)\right\}$, is then $\tilde{J}(\theta )=\mathbb{E}\left\{\text{(}{l}_1\circ F_{\theta }\left(Z^{\prime }\right)\right\}=J(\theta )+\text{Δ}(\theta )$ where $\text{Δ}(\theta )=\mathbb{E}\left[\text{Var}\left\{F_{\theta }\left(Z^{\prime }\right)|Z\right\}\right]$. The minimizer of $\tilde{J}(\theta )$ is not necessarily θ* unless further conditions are imposed (e.g., Var{F_θ(Z′)|Z} is independent of θ, which is often not the case in our setting).

The high level idea of coupled estimation is to first estimate f_θ for each θ, denoted by ${\widehat{f}}_{\theta }$, and then estimate θ* by the plug-in estimator, ${\text{argmin}}_{\theta }{\mathbb{P}}_n\{(l_1\circ {\widehat{f}}_{\theta })(Z)\}$. A standard empirical risk minimization can be applied to obtain a consistent estimator for f_θ, e.g., ${\widehat{f}}_{\theta }={\text{argmin}}_{g\in 𝒢}{\mathbb{P}}_n\left[l_2\text{{(}{F}_{\theta }\left(Z^{\prime }\right),g(Z)\text{}]}\right.$ for some loss function $l_2:ℝ\times ℝ\to {ℝ}^{+}$ and a function space, $𝒢$ to approximate f_θ. We call the estimator coupled because the objective function (i.e., ${\mathbb{P}}_n\{(l_1\circ {\widehat{f}}_{\theta })(Z)\}$) involves ${\widehat{f}}_{\theta }$ which itself is an minimizer of another objective function (i.e., ${\mathbb{P}}_n{l}_2\left(\left(F_{\theta }\left(Z^{\prime }\right),g(Z)\right)\right.$) for each θ.

4.2. Relative value function estimator.

Recall the doubly robust estimator requires an estimator of U^π. It is enough to learn one specific version of Q^π. More specifically, define a shifted value function by ${\tilde{Q}}^{\pi }(s,a)=Q^{\pi }(s,a)-Q^{\pi }\left(s^{\ast },a^{\ast }\right)$ for some specific state-action pair (s*, a*). By restricting to Q(s*, a*) = 0, the solution of Bellman equations (3.2) is unique and given by $\{{\eta }^{\pi },{\tilde{Q}}^{\pi }\}$. Below we derive a coupled estimator for the shifted value function, ${\tilde{Q}}^{\pi }$, using the coupled estimation framework in Section 4.1.

Let Z_t = (S_t, A_t, S_t+1) be the transition sample at time t. For a given (η, Q) pair, let

$${\delta }^{\pi }\left(Z_t;\eta ,Q\right)=R_{t+1}+\sum _{a^{\prime }}\pi \left(a^{\prime }|S_{t+1}\right)Q\left(S_{t+1},a\right)-Q\left(S_t,A_t\right)-\eta$$ (4.2)

be the so-called temporal difference (TD) error. The Bellman equation then becomes $\mathbb{E}\left[{\delta }^{\pi }\left(Z_t;\eta ,Q\right)|S_t=s,A_t=a\right]=0$ for all state-action pair, (s, a). As a result, we have

$$\{{\eta }^{\pi },{\tilde{Q}}^{\pi }\}\in \underset{\eta ,Q}{\text{argmin}}\mathbb{E}\left[\frac{1}{T}\sum _{t=1}^T{\left(\mathbb{E}\left[{\delta }^{\pi }\left(Z_t;\eta ,Q\right)|S_t,A_t\right]\right)}^2\right].$$

Note that above we choose the squared loss for simplicity; a general loss function can also be applied. We see that it fits in the coupled estimation framework presented in the previous section. In particular, ${\theta }^{\ast }=\{{\eta }^{\pi },{\tilde{Q}}^{\pi }\}$ and f_θ becomes the Bellman error, i.e., $\mathbb{E}\left[{\delta }^{\pi }\left(Z_t;\eta ,Q\right)|S_t=\cdot ,A_t=\cdot \right]$. The above characterization involves the average reward, η^π. Thus in the process of obtaining an estimator of the relative value function, we will also obtain an estimator of the average reward. See the end of this subsection for discussion.

We use $ℱ$ and $𝒢$ to denote two classes of functions of state-action. We use $ℱ$ to model the shifted value function, ${\tilde{Q}}^{\pi }$, and thus require f(s^∗, a^∗) = 0 for all $f\in ℱ$. We use $𝒢$ to approximate the conditional mean of the Bellman error. In addition, $J_1:ℱ\to {ℝ}^{+}$ and $J_2:𝒢\to {ℝ}^{+}$ are two regularizers that measure the complexities of these two functional classes respectively. Given tuning parameters (λ_n, μ_n), the coupled estimator, denoted by $({\widehat{\eta }}_n^{\pi },{\widehat{Q}}_n^{\pi })$, is obtained by solving

$$({\tilde{\eta }}_n^{\pi },{\widehat{Q}}_n^{\pi })=\underset{(\eta ,Q)\in ℝ\times ℱ}{\text{argmin}}{\mathbb{P}}_n\left[\frac{1}{T}\sum _{t=1}^T{\widehat{g}}_n^{\pi }{\left(S_t,A_t;\eta ,Q\right)}^2\right]+{\lambda }_n{J}_1^2(Q),$$ (4.3)

where ${\widehat{g}}_n^{\pi }(\cdot ,\cdot ;\eta ,Q)$ is the projected Bellman error at (η, Q):

$${\widehat{g}}_n^{\pi }(\cdot ,\cdot ;\eta ,Q)=\underset{g\in 𝒢}{\text{argmin}}{\mathbb{P}}_n\left[\frac{1}{T}\sum _{t=1}^T{\left({\delta }^{\pi }\left(Z_t;\eta ,Q\right)-g\left(S_t,A_t\right)\right)}^2\right]+{\mu }_n{J}_2^2(g).$$ (4.4)

Given the estimator of the (shifted) relative value function, ${\widehat{Q}}_n^{\pi }$, we form the estimator of U^π by ${\widehat{U}}_n^{\pi }\left(s,a,s^{\prime }\right)={\sum }_{a^{\prime }}\pi \left(a^{\prime }|s^{\prime }\right){\widehat{Q}}_n^{\pi }\left(s^{\prime },a^{\prime }\right)-{\widehat{Q}}_n^{\pi }(s,a)$.

Throughout this paper, we use tuning parameters, (λ_n, μ_n), that do not depend on the policy. In the setting where the policy class is highly complex and the corresponding relative value functions are very different, it could be beneficial to select the tuning parameters locally at a cost of higher computation burden.

Recall that the goal here is to estimate relative value function, U^π, and then plug ${\widehat{U}}_n^{\pi }$ in the doubly robust estimator (3.6). The above ${\tilde{\eta }}_n^{\pi }$ is only used to help estimate the relative function. In fact, Liao, Klasnja and Murphy (2019) proposed using ${\tilde{\eta }}_n^{\pi }$ to estimate the average reward. The advantage of our doubly robust estimator (3.6), ${\widehat{\eta }}_n^{\pi }$, over ${\tilde{\eta }}_n^{\pi }$ is that the consistency of ${\widehat{\eta }}_n^{\pi }$ is guaranteed as long as at least one of the nuisance function is estimated consistently (Theorem 3.2).

4.3. Ratio function estimator.

Below we derive the estimator for the ratio function, ω^π using the coupled estimation framework. In particular we estimate a scaled version of the ratio function (denoted by e^π below) and then convert this back to an estimator of ω^π. To estimate e^π, we first construct a new MDP and estimate the relative value function for this new MDP (denoted by H^π) using the coupled estimation framework. The estimator of e^π is then derived from the estimator of H^π.

We start with introducing e^π:

$$e^{\pi }(s,a)=\frac{{\omega }^{\pi }(s,a)}{{\sum }_{\tilde{s},\tilde{a}}{\omega }^{\pi }(\tilde{s},\tilde{a})d^{\pi }(\tilde{s})\pi (\tilde{a}|\tilde{s})}.$$ (4.5)

By definition, ${\sum }_{s,a}{e}^{\pi }(s,a)d^{\pi }(s)\pi (a|s)=1$. If we were to replace the reward function in our MDP by 1−e^π(s, a), then the “average reward” of π in this new MDP is constant and equal to zero under Assumption 1 (i.e., ${\sum }_{s,a}\left\{1-e^{\pi }(s,a)\right\}{d}^{\pi }(s)\pi (a|s)=0$). The “relative value function” of policy π under the new MDP is,

$$H^{\pi }(s,a)=\underset{t^{\ast }\to \infty }{\text{lim}}\frac{1}{t^{\ast }}\sum _{t=1}^{t^{\ast }}{\mathbb{E}}_{\pi }\left[\sum _{k=1}^t\left\{1-e^{\pi }\left(S_k,A_k\right)\right\}|S_1=s,A_1=a\right].$$ (4.6)

Note that H^π is well-defined under Assumption 1. Furthermore, consider the following Bellman equation for the new MDP:

$${\mathbb{E}}_{\pi }\left\{1-e^{\pi }\left(S_t,A_t\right)+H\left(S_{t+1},A_{t+1}\right)|S_t=s,A_t=a\right\}=H(s,a).$$ (4.7)

Note that since the “average reward” in the new MDP is zero, the above equation only involves H. The set of solutions of (4.7) can be shown to be $\left\{H:H=H^{\pi }+c\mathbf{1},c\in ℝ\right\}$.

Below we construct a coupled estimator for a shifted version of H^π, i.e., ${\tilde{H}}^{\pi }=H^{\pi }-H^{\pi }\left(s^{\ast },a^{\ast }\right)$. Recall Z_t = (S_t, A_t, S_t+1, R_t+1) is the transition sample at time t. For a given state-action function, H, let ${\text{Δ}}^{\pi }\left(Z_t;H\right)=1-H\left(S_t,A_t\right)+{\sum }_{a^{\prime }}\pi \left(a^{\prime }|S_{t+1}\right)H\left(S_{t+1},a^{\prime }\right)$. As a result of the above Bellman-like equation and the orthogonality property (3.4), we know that

$${\tilde{H}}^{\pi }\in \underset{H}{\text{argmin}}\mathbb{E}\left[\frac{1}{T}\sum _{t=1}^T{\left(\mathbb{E}\left[{\text{Δ}}^{\pi }\left(Z_t;H\right)|S_t,A_t\right]\right)}^2\right].$$

Now it can be seen that the estimation of ${\tilde{H}}^{\pi }$ fits into the coupled estimation framework (4.1), i.e., ${\theta }^{\ast }={\tilde{H}}^{\pi }$ and f_θ is $\mathbb{E}\left[{\text{Δ}}^{\pi }\left(Z_t;H\right)|S_t=\cdot ,A_t=\cdot \right]$. With a slight abuse of notation, we use $ℱ$ to approximate ${\tilde{H}}^{\pi }$ and $𝒢$ to form the approximation of $\mathbb{E}\left[{\text{Δ}}^{\pi }\left(Z_t;H\right)|S_t=\cdot ,A_t=\cdot \right]$. The coupled estimator, ${\widehat{H}}_n^{\pi }$, is then found by solving

$${\widehat{H}}_n^{\pi }=\underset{H\in ℱ}{\text{argmin}}{\mathbb{P}}_n\left[\frac{1}{T}\sum _{t=1}^T{\widehat{g}}_n^{\pi }{\left(S_t,A_t;H\right)}^2\right]+{\lambda }_n^{\prime }{J}_1^2(H),$$ (4.8)

where for any $H\in ℱ$, ${\widehat{g}}_n^{\pi }(\cdot ,\cdot ;H)$ solves

$${\widehat{g}}_n^{\pi }(\cdot ,\cdot ;H)=\underset{g\in 𝒢}{\text{argmin}}{\mathbb{P}}_n\left[\frac{1}{T}\sum _{t=1}^T{\left({\text{Δ}}^{\pi }\left(Z_t;H\right)-g\left(S_t,A_t\right)\right)}^2\right]+{\mu }_n^{\prime }{J}_2^2(g).$$ (4.9)

Recall that e^π can be written in terms of H^π by (4.7); that is, re-arranging terms,

$${\mathbb{E}}_{\pi }\left\{1-H\left(S_t,A_t\right)+H\left(S_{t+1},A_{t+1}\right)|S_t=s,A_t=a\right\}=e^{\pi }(s,a).$$

Thus given the estimator, ${\widehat{H}}_n^{\pi }$, we estimate e^π by ${\widehat{e}}_n^{\pi }(s,a)={\widehat{g}}_n(s,a;{\widehat{H}}_n^{\pi })$. By the definition of ω^π, we have $\mathbb{E}[(1/T){\sum }_{t=1}^T{\omega }^{\pi }\left(S_t,A_t\right)]=1$. Since e^π is a scaled version of ω^π up to a constant, we finally construct the estimator for ratio, ω^π, by scaling ${\widehat{e}}^{\pi }$, that is,

$${\widehat{\omega }}_n^{\pi }(s,a)={\widehat{e}}_n^{\pi }(s,a)/{\mathbb{P}}_n[(1/T)\sum _{t=1}^T{\widehat{e}}_n^{\pi }\left(S_t,A_t\right)],\forall (s,a)\in 𝒮\times 𝒜.$$ (4.10)

Remark 3.

The above ratio function estimator was developed by Liao, Klasnja and Murphy (2019). In this paper we, for the first time, derive a finite-sample error bound for this ratio function estimator, uniformly over the policy class (Theorem B.2 in the appendix). This is the key element in establishing the finite-sample bound regret bound for the estimated optimal policy.

Remark 4.

Our ratio function estimator is different from most in the existing literature, such as Liu et al. (2018); Uehara and Jiang (2019); Nachum et al. (2019); Zhang et al. (2020), which are obtained by min-max based estimating methods. For example, Liu et al. (2018) aimed to estimate the ratio between stationary distribution induced by a known, Markovian time-stationary behavior policy and target policy, which is then used to estimate the average reward of a given policy. This is not suitable for the setting where the behavior policy is history dependent. Uehara and Jiang (2019) estimated the ratio, ω^π(s, a), based on the observation that for every state-action function f,

$$\mathbb{E}\left[\frac{1}{T}\sum _{t=1}^T\left({\omega }^{\pi }\left(S_t,A_t\right)\sum _{a^{\prime }}\pi \left(a^{\prime }|S_{t+1}\right)f\left(S_{t+1},a\right)-{\omega }^{\pi }\left(S_t,A_t\right)f\left(S_t,A_t\right)\right)\right]=0,$$

with the restriction that $\mathbb{E}[\frac{1}{T}{\sum }_{t=1}^T{\omega }^{\pi }\left(S_t,A_t\right)]=1$. Then they constructed their estimator by solving the empirical version of the following min-max optimization problem:

$$\underset{w\in \text{Δ}}{\text{min}}\underset{f\in {ℱ}^{\prime }}{\text{max}}{\mathbb{E}}^2\left[\frac{1}{T}\sum _{t=1}^T\left(\omega \left(S_t,A_t\right)\sum _{a^{\prime }}\pi \left(a^{\prime }|S_{t+1}\right)f\left(S_{t+1},a\right)-\omega \left(S_t,A_t\right)f\left(S_t,A_t\right)\right)\right],$$

where Δ is a simplex space and ${ℱ}^{\prime }$ is a set of discriminator functions. This method minimizes the upper bound of the bias of their average reward estimator if the state-action value function is contained in ${ℱ}^{\prime }$. They proved consistency of their ratio and average reward estimators in the parametric setting, that is, where ω^π(S_t, A_t) can be modelled parametrically and ${ℱ}^{\prime }$ is a finite dimensional space. Subsequently Zhang et al. (2020) developed a general min-max based estimator by considering variational f-divergence, which subsumes the case in Uehara and Jiang (2019). Unfortunately, there are no error bounds guarantee for ratio function estimators developed in Uehara and Jiang (2019) and Zhang et al. (2020). Our ratio estimator appears closely related to the estimation developed by Nachum et al. (2019) as they also formulated the ratio estimator as a minimizer of a loss function. However, relying on the Fenchel’s duality theorem, they still use the min-max based method to estimate the ratio. Furthermore, their method cannot be applied in average reward settings. Instead of using min-max based estimators, we use coupled estimation. This will facilitate the derivation of estimation error bounds as will be seen below. We will derive the estimation error of the ratio function, which will enable us to provide a strong theoretical guarantee, and finally demonstrate the efficiency of our average reward estimator without imposing restrictive parametric assumptions on the nuisance function estimations, see Section 5 below.

5. Theoretical Results.

5.1. Regret bound.

In this section, we provide a finite sample bound on the regret of ${\widehat{\pi }}_n$ defined in (2.4), i.e., the difference between the optimal average reward in the policy class, Π, and the average reward of the estimated policy, ${\widehat{\pi }}_n$.

Consider a state-action function, f(s, a). Let $ℐ$ be the identity operator, i.e., $ℐ(f)=f$. Denote the conditional expectation operator by ${𝒫}^{\pi }f:(s,a)\mapsto {\mathbb{E}}_{\pi }\left[f\left(S_{t+1},A_{t+1}\right)|S_t=\right.\left.s,A_t=a\right]$. Let the expectation under stationary distribution induced by π be μ^π(f) =∫f(s, a) μ^π (ds, da). Denote by ∥ · ∥_tv the total variation distance between two probability measures. For a function g(s, a, s′), define $\parallel g{\parallel }^2=\mathbb{E}\left\{(1/T){\sum }_{t=1}^T{g}^2\left(S_t,A_t,S_{t+1}\right)\right\}$. For a set $𝒳$ and M > 0, let $ℬ(X,M)$ be the class of bounded functions on $𝒳$ such that ∥f∥_∞ ≤ M. Denote by $N(ϵ,ℱ,\parallel \cdot \parallel )$ the ϵ-covering number of a set of functions, $ℱ$, with respect to the norm, ∥· ∥.

We make use of the following assumption on Π.

Assumption 3.

The policy class, $\text{Π}=\left\{{\pi }_{\theta }:\theta \in \text{Θ}\subset {ℝ}^p\right\}$, satisfies:

• (3–1)$\text{Θ}\subset {ℝ}^p$ is compact and let $\text{diam}(\text{Θ})={\text{sup}}_{{\theta }_1,{\theta }_2\in \text{Θ}}{‖{\theta }_1-{\theta }_2‖}_2$.
• (3–2)
  There exists L_Θ > 0, such that for θ₁, θ₂ ∈ Θ and for all $(s,a)\in 𝒮\times 𝒜$, the following holds

  $$\left|{\pi }_{{\theta }_1}(a|s)-{\pi }_{{\theta }_2}(a|s)\right|\le L_{\text{Θ}}{‖{\theta }_1-{\theta }_2‖}_2.$$
• (3–3)
  There exists constants C₀ > 0 and 0 ≤ β < 1, such that for every π ∈ Π, the following hold for all t ≥ 1:

  $${‖P^{\pi }\left(S_t=\cdot |S_1=s\right)-d^{\pi }(\cdot )‖}_{\text{tv}}\le C_0{\beta }^t,$$ (5.1)

  $$‖{\left({𝒫}^{\pi }\right)}^{t}f-{\mu }^{\pi }(f)‖\le C_0\parallel f\parallel {\beta }^t.$$ (5.2)

Remark 5.

The Lipschitz property of the policy class (3–2) is used to control the complexity of nuisance function induced by Π, that is, {U^π(·,·,·) : π ∈ Π} and {ω^π(·,·) : π ∈ Π}. This is commonly assumed in the finite-time horizon problems (e.g., Zhou et al. (2017)). Assumptions (3–1) and (3–2) can be easily satisfied by many policy classes such the one we used in Section 7. Our analysis can be extended to more general policy classes if a similar complexity property holds for these two nuisance function classes. Intuitively the constant β in the assumption (3–3) relates to the “mixing time” of the Markov chain induced by π ∈ Π. A similar assumption was used by Van Roy (1998); Liao, Klasnja and Murphy (2019) in average reward setting. Specifically, Equation (5.1) in Assumption (3–3) is used to show that two nuisance functions U^π and ω^π are Lipschitz continuous with respect to the policy parameter θ so that we can quantify their estimation error uniformly over the policy class. See Lemma C.1 of Supplementary Material for more details. Equation (5.2) in Assumption (3–3) basically requires an exponential convergence rate of the policy induced Markov chain to the stationary distribution in terms of the expectation under the L₂-norm with respect to the data generating process. This assumption, together with Assumption (5–3) stated below is used to guarantee the Bellman operator for U^π based on Equation (3.2) (or a similar quantity related to the ratio function estimation defined in Lemma B.4 of Supplementary Material) is well-posed in the sense of L₂-norm with respect to the data generating process so as to derive their estimation errors. See Lemma B.5 of Liao, Klasnja and Murphy (2019) and Lemma B.4 of Supplementary Material for more details.

Recall that we use the same pair of function classes $(ℱ,𝒢)$ in the coupled estimation for both U^π and ω^π. We make the following assumptions on $(ℱ,𝒢)$.

Assumption 4.

The function classes, $(ℱ,𝒢)$, satisfy the following:

• (4–1)$ℱ\subset ℬ\left(𝒮\times 𝒜,F_{\text{max}}\right)$ and $𝒢\subset ℬ\left(𝒮\times 𝒜,G_{\text{max}}\right)$ (4–2) f(s^∗, a^∗) = 0, $f\in ℱ$.
• (4–3)The regularization functionals, J₁ and J₂, are pseudo norms and induced by the inner products J₁(·,·) and J₂(·,·), respectively.
• (4–4)
  Let ${ℱ}_M=\left\{f\in ℱ:J_1(f)\le M\right\}$ and ${𝒢}_M=\left\{g\in 𝒢:J_2(g)\le M\right\}$. There exists C₁ and α ∈ (0, 1) such that for any ϵ, M > 0,

  $$\text{max}\left\{\text{log}N\left(ϵ,{𝒢}_M,\parallel \cdot {\parallel }_{\infty }\right),\text{log}N\left(ϵ,{ℱ}_M,\parallel \cdot {\parallel }_{\infty }\right)\right\}\le C_1{\left(\frac{M}{ϵ}\right)}^{2\alpha }$$

Remark 6.

The boundedness assumption on $ℱ$ and $𝒢$ are used to simplify the analysis and can be relaxed by truncating the estimators. We restrict f(s*, a*) = 0 for all $f\in ℱ$ because $ℱ$ is used to model ${\tilde{Q}}^{\pi }$ and ${\tilde{H}}^{\pi }$, which by definition satisfies ${\tilde{Q}}^{\pi }\left(s^{\ast },a^{\ast }\right)=0$ and ${\tilde{H}}^{\pi }\left(s^{\ast },a^{\ast }\right)=0$. In Section 6, we show how to shape an arbitrary kernel function to ensure this is satisfied automatically when $ℱ$ is RKHS. The complexity assumption (4–4) on $ℱ$ and $𝒢$ are satisfied for common function classes, for example RKHS and Sobolev spaces (Steinwart and Christmann, 2008; Györfi et al., 2006). Taking the Sobolev spaces as an example, the entropy exponent α will be $p/\tilde{q}$, where p is the dimension of state-variables and $\tilde{q}$ is the number of continuous derivatives possessed by the functions in the corresponding space. Assumption (4–4) is imposed to control the estimation error for two nuisance functions.

We now introduce the assumption that is used to bound the estimation error of value function uniformly over the policy class. Define the projected Bellman error operator:

$$g_{\pi }^{\ast }(\cdot ,\cdot ;\eta ,Q):=\underset{g\in 𝒢}{\text{argmin}}\mathbb{E}\left[\frac{1}{T}\sum _{t=1}^T{\left\{{\delta }^{\pi }\left(Z_t;\eta ,Q\right)-g\left(S_t,A_t\right)\right\}}^2\right]$$

where δ^π is given in (4.2).

Assumption 5.

The triplet, $(\text{Π},ℱ,𝒢)$, satisfies the following:

• (5–1)${\tilde{Q}}^{\pi }(\cdot ,\cdot )\in ℱ$ for π ∈ Π and ${\text{sup}}_{\pi \in \text{Π}}{J}_1({\tilde{Q}}^{\pi })<\infty$.
• (5–2)$0\in 𝒢$.
• (5–3)There exits κ > 0, such that $\text{inf}\left\{‖g_{\pi }^{\ast }(\cdot ,\cdot ;\eta ,Q)‖:‖\mathbb{E}\left[{\delta }^{\pi }\left(Z_t;\eta ,Q\right)|S_t=\cdot ,A_t=\cdot \right]‖=\right.\left.1,|\eta |\le R_{\text{max}},Q\in ℱ,\pi \in \text{Π}\right\}\ge \kappa$.
• (5–4)There exists two constants C₂, C₃ such that $J_2\left\{g_{\pi }^{\ast }(\cdot ,\cdot ;\eta ,Q)\right\}\le C_2+C_3{J}_1(Q)$ holds for all $\eta \in ℝ$, $Q\in ℱ$ and π ∈ Π.

Remark 7.

Assumption (5–1) basically assumes that the non-parametric function class $ℱ$ can model ${\tilde{Q}}^{\pi }$ correctly, which is mild. Note that in the coupled estimator of ${\tilde{Q}}^{\pi }$, we do not require the much stronger condition that the Bellman error for every tuple of (η, Q, π) is correctly modeled by $𝒢$. In other words, $\mathbb{E}\left[{\delta }^{\pi }\left(Z_t;\eta ,Q\right)|S_t=\cdot ,A_t=\cdot \right]$ is not required to belong to $𝒢$. Instead, the combination of conditions (5–2) and (5–3) is enough to guarantee the consistency of the coupled estimator (recall that the Bellman error is zero at $\{{\eta }^{\pi },{\tilde{Q}}^{\pi }\}$). The last condition (5–4) essentially requires the transition matrix is sufficiently smooth so that the complexity of the projected Bellman error, $J_2\left\{g_{\pi }^{\ast }(\cdot ,\cdot ;\eta ,Q)\right\}$, can be controlled by J₁(Q), the complexity of Q (see Farahmand et al. (2016) for an example).

A similar set of conditions are employed to bound the estimation of ratio function. For π ∈ Π and $H\in ℱ$, define the projected error:

$$g_{\pi }^{\ast }(\cdot ,\cdot ;H)=\underset{g\in 𝒢}{\text{argmin}}\mathbb{E}\left[\frac{1}{T}\sum _{t=1}^T{\left\{{\text{Δ}}^{\pi }\left(Z_t;H\right)-g\left(S_t,A_t\right)\right\}}^2\right]$$

where, as before, ${\text{Δ}}^{\pi }\left(Z_t;H\right)=1-H\left(S_t,A_t\right)+{\sum }_{a^{\prime }}\pi \left(a^{\prime }|S_{t+1}\right)H\left(S_{t+1},a^{\prime }\right)$.

Assumption 6.

The triplet, $(\text{Π},ℱ,𝒢)$, satisfies the following:

• (6–1)For π ∈ Π, ${\tilde{H}}^{\pi }(\cdot ,\cdot )\in ℱ$, and ${\text{sup}}_{\pi \in \text{Π}}{J}_1({\tilde{H}}^{\pi })<\infty$.
• (6–2)$e^{\pi }(\cdot ,\cdot )\in 𝒢$, for π ∈ Π.
• (6–3)There exits κ′ > 0, such that $\text{inf}\{g_{\pi }^{\ast }(\cdot ,\cdot ;H)-g_{\pi }^{\ast }(\cdot ,\cdot ;{\tilde{H}}^{\pi }):(ℐ-{𝒫}^{\pi })(H-{\tilde{H}}^{\pi })=\}1,H\in ℱ,\pi \in \text{Π}\}\ge {\kappa }^{\prime }$.
• (6–4)There exists two constants $C_2^{\prime }$, $C_3^{\prime }$ such that $J_2\left\{g_{\pi }^{\ast }(\cdot ,\cdot ;H)\right\}\le C_2^{\prime }+C_3^{\prime }{J}_1(H)$ holds for $H\in ℱ$ and π ∈ Π.

Remark 8.

The interpretation of Assumption 6 is similar to that of Assumption 5. Specifically, Assumption (6–1) basically assumes the non-parametric function class $ℱ$ can model ${\tilde{H}}^{\pi }$ correctly. As in the case of estimation of relative value function, we do not require the correct modelling of $\mathbb{E}\left[{\text{Δ}}^{\pi }\left(Z_t;H\right)|S_t=\cdot ,A_t=\cdot \right]$ for every $(\pi ,H)\in \text{Π}\times ℱ$ but instead only assume Assumptions (6–2) and (6–3) hold. A major difference between Assumption 5 and 6 is that (5–2) is now replaced by (6–2). This is because according to the Bellman-like equation (4.7), we have $\mathbb{E}[{\text{Δ}}^{\pi }(Z_t;{\tilde{H}}^{\pi })|S_t=s,A_t=a]=e^{\pi }(s,a)$.

Theorem 5.1.

Suppose Assumptions 1 to 6 hold. Let ${\widehat{\pi }}_n$ be the estimated policy (2.3) in which the nuisance functions are estimated with tuning parameters ${\mu }_n={\lambda }_n={\mu }_n^{\prime }={\lambda }_n^{\prime }=L{n}^{-1/(1+\alpha )}$, for some constant L > 0. Define ${\beta }_k=\frac{1}{1+\alpha }\left\{1-(1-\alpha )2^{-k+1}\right\}$. Fix any integer k ≥ 2, δ ∈ (0, 1) and sufficiently large n. With probability at least 1 − δ, we have

$$\text{Regret}\left({\widehat{\pi }}_n\right)\le C(\delta )\left(p^{1/2}{n}^{-1/2}+p{n}^{-{\beta }_k}\right),$$

where C(δ) is a function of k, L, F_max, G_max, L_Θ, diam(Θ), ${\text{sup}}_{\pi \in \text{Π}}{J}_1({\tilde{H}}^{\pi })$, ${\text{sup}}_{\pi \in \text{Π}}{J}_1({\tilde{Q}}^{\pi })$, ${\text{sup}}_{\pi \in \text{Π}}‖{\omega }^{\pi }‖$, α, constants $\left\{C_0,C_1,C_2,C_3,C_2^{\prime },C_3^{\prime }\right\}$, κ, κ′, β, p_min and ${\Vert \frac{d_{T+1}}{d_D}\Vert }_{\infty }$.

Remark 9.

Recall that p is the number of parameters in the policy, α is given in (4–4), and n is the number of trajectories in the data. Theorem 5.1 shows that when the tuning parameters are of the order O(n^−1/(1+α)), the regret of the estimated policy is $O\left(p^{1/2}{n}^{-1/2}+p{n}^{-{\beta }_k}\right)$. The leading term (in terms of n), $O(\sqrt{p/n})$, corresponds to the regret of an estimated policy as if the nuisance functions are known beforehand. The second term is due to the estimation error of nuisance functions. In particular, we show in Theorem B.1 in Section B of the appendix that the uniform estimation error of the relative value function is of O(pn^−1/(1+α)) and in Theorem B.2 in the same section that the uniform estimation error of ratio is of $O\left(p{n}^{-{\beta }_k}\right)$ (see the remark after Theorem B.2 for why the rate depends on k). Note that the error of ratio is the dominant term as β_k < 1/(1 + α) and β_k can be chosen arbitrarily close to $\frac{1}{1+\alpha }$ by choosing a sufficiently large k. Therefore the proposed ratio estimator can achieve the near-optimal nonparametric convergence rate. See the proof of Theorem B.2 in Section B.1 of the appendix for more details. To the best of our knowledge, this is the first result that characterizes the regret of the estimated optimal in-class policy in the infinite horizon setting.

5.2. Asymptotic results.

In this section, we prove that the average reward for our estimator of the optimal policy converges to the optimal average reward at a parametric rate (i.e., $\sqrt{n}$). Recall ϕ^π(D) is the efficient influence function of η^π given in Theorem 3.1.

Theorem 5.2.

Suppose Assumptions 1 to 6 hold. For each n ≥ 1, let ${\widehat{\eta }}_n^{\pi }$ be the doubly robust estimator defined in (3.6) and ${\widehat{\pi }}_n$ be the estimated policy defined in (2.3) with tuning parameters ${\mu }_n={\lambda }_n={\mu }_n^{\prime }={\lambda }_n^{\prime }=L{n}^{-1/(1+\alpha )}$, for some constant L > 0. Then as n → ∞,

1. $\left\{\sqrt{n}\left({\widehat{\eta }}_n^{\pi }-{\eta }^{\pi }\right):\pi \in \text{Π}\right\}\Rightarrow \mathbb{G}(\pi )$ in l^∞(Π) where $\mathbb{G}(\pi )$ is a zero mean Gaussian Process with covariance function $𝒞:\text{Π}\times \text{Π}\to ℝ$, $𝒞\left({\pi }_1,{\pi }_2\right)=\mathbb{E}\left\{{\varphi }^{{\pi }_1}(D){\varphi }^{{\pi }_2}(D)\right\}$.

2. $\sqrt{n}({\widehat{\eta }}_n^{{\widehat{\pi }}_n}-{\text{sup}}_{\pi \in \text{Π}}{\eta }^{\pi })\Rightarrow {\text{sup}}_{\pi \in {\text{Π}}_{\text{max}}}\mathbb{G}(\pi )$, where $\mathbb{G}(\pi )$ is the Gaussian Process defined above and ${\text{Π}}_{\text{max}}={\text{argmax}}_{\pi \in \text{Π}}{\eta }^{\pi }$ is the set of policies that maximize the average reward in Π.

Remark 10.

The first result shows that the estimated average reward by the doubly robust estimator reaches the semiparametric efficiency bound when we plug in the estimator for the two nuisance functions. The double robustness structure ensures that the estimation error of nuisance functions is only of lower order and does not impact the asymptotic variance of the estimated average reward. The second result shows the asymptotic of the estimated optimal value, ${\widehat{\eta }}_n^{{\widehat{\pi }}_n}$, converges to the maximum of the Gaussian process at the optimal policies. When there is a unique optimal policy ${\pi }^{\ast }={\text{argmax}}_{\pi \in \text{Π}}{\eta }^{\pi }$, we have $\sqrt{n}({\widehat{\eta }}_n^{{\widehat{\pi }}_n}-{\eta }^{{\pi }^{\ast }})$ weakly converges to a Gaussian distribution. Estimating the limiting distribution could be challenging (especially when there exists non-unique policies) and is left for future work. Alternatively one can consider resampling-based method to construct confidence interval for sup_π η^π (see the recent work by Wu and Wang (2020) in single-stage problem).

6. Practical Implementation.

In this section, we describe an algorithm to estimate an in-class optimal policy based on our efficient average reward estimator ${\widehat{\eta }}_n^{\pi }$. Without loss of generality, we consider a binary-action setting, i.e., $𝒜=\{0,1\}$, and the following stochastic parametrized policy class Π indexed by θ:

$$\text{Π}=\left\{\pi |\pi (1|s,\theta )=\frac{\text{exp}\left(s^T\theta \right)}{1+\text{exp}\left(s^T\theta \right)},\parallel \theta {\parallel }_{\infty }\le c,\theta \in {ℝ}^p\right\},$$

for some pre-specified constant c > 0. Note that other link functions such as the probit function might be used here instead. Here ∥ · ∥_∞ refers to sup-norm in Euclidean space. We fix c = 10 throughout our paper. In addition, we set $ℱ$ and $𝒢$ in the estimation of both value and ratio functions to be Reproducing Kernel Hilbert Spaces (RKHSs) associated with Gaussian kernels because of the representer theorem and the property of universal consistency.

The constraint on θ, ∥θ∥_∞ ≤ c, is used to maintain sufficient stochasticity in our learned policy. The stochasticity facilitates the use of ${\widehat{\pi }}_n$ as a “warm start” policy for use by an online algorithm with future individuals. A nice side effect is that the restriction on θ provides a computational stability and can avoid degenerative cases in policy optimization similar to that when using logistic regression in classification problems (Friedman, Hastie and Tibshirani, 2001). As discussed in the introduction, we consider the simple policy class Π instead of nonparametric models such as neural networks or tree-based models mainly due to the concern of overfitting. In the batch setting, data are limited and often noisy. Using flexible function classes for modeling the policy may lead to overfitting and thus the variance of the resulting policy could be very large. The use of a simple policy class can reduce the variance while it may induce some possible bias. In addition, interpretability is critical in our batch policy learning problem. The interpretability of decision tree models are often not very stable, whereas neural networks are not very interpretable. Therefore we prefer using this simple policy class Π.

To obtain ${\widehat{\pi }}_n\in \text{Π}$, we solve a multi-level optimization problem (6.1)–(6.5). Recall a multi-level optimization problem (Richardson, 1995) is a optimization problems in which the feasible set is implicitly determined by a sequence of nested optimization problems. It typically consists of an upper level optimization task that represents the objective function, and a series of (possibly nested) lower level optimization tasks that represents the feasible set.

Upper level optimization task:

$$\underset{\pi \in \text{Π}}{\text{max}}\frac{{\mathbb{P}}_n\left\{(1/T){\sum }_{t=1}^T{\widehat{\omega }}_n^{\pi }\left(S_t,A_t\right)\left[R_{t+1}+{\widehat{U}}_n^{\pi }\left(S_t,A_t,S_{t+1}\right)\right]\right\}}{{\mathbb{P}}_n\left\{(1/T){\sum }_{t=1}^T{\widehat{\omega }}_n^{\pi }\left(S_t,A_t\right)\right\}}$$ (6.1)

Lower level optimization task 1:

$$({\widehat{\eta }}_n^{\pi },{\widehat{Q}}_n^{\pi })=\underset{(\eta ,Q)\in ℝ\times ℱ}{\text{argmin}}{\mathbb{P}}_n\left[\frac{1}{T}\sum _{t=1}^T{\left[{\widehat{g}}_n^{\pi }\left(S_t,A_t;\eta ,Q\right)\right]}^2\right]+{\lambda }_n{J}_1^2(Q)$$ (6.2)

$$\text{s.t}.{\widehat{g}}_n^{\pi }(\cdot ,\cdot ;\eta ,Q)={\text{argmin}}_{g\in 𝒢}{\mathbb{P}}_n\left[\frac{1}{T}\sum _{t=1}^T{\left({\delta }^{\pi }\left(Z_t;\eta ,Q\right)-g\left(S_t,A_t\right)\right)}^2\right]+{\mu }_n{J}_2^2(g)$$ (6.3)

Lower level optimization task 2:

$${\widehat{H}}_n^{\pi }(\cdot ,\cdot )=\underset{H\in ℱ}{\text{argmin}}{\mathbb{P}}_n\left[\frac{1}{T}\sum _{t=1}^T{\left[{\widehat{g}}_n^{\pi }\left(S_t,A_t;H\right)\right]}^2\right]+{\lambda }_n^{\prime }{J}_1^2(H)$$ (6.4)

$$\text{s.t}.{\widehat{g}}_n^{\pi }(\cdot ,\cdot ;H)=\underset{g\in 𝒢}{\text{argmin}}{\mathbb{P}}_n\left[\frac{1}{T}\sum _{t=1}^T{\left({\text{Δ}}^{\pi }\left(Z_t;H\right)-g\left(S_t,A_t\right)\right)}^2\right]+{\mu }_n^{\prime }{J}_2^2(g).$$ (6.5)

As a reminder, recall that in Section 4 we have defined

$${\delta }^{\pi }\left(Z_t;\eta ,Q\right)=R_{t+1}+\sum _{a^{\prime }}\pi \left(a^{\prime }|S_{t+1}\right)Q\left(S_{t+1},a\right)-Q\left(S_t,A_t\right)-\eta ,$$

$${\widehat{U}}_n^{\pi }\left(S_t,A_t,S_{t+1}\right)=\sum _{a\in 𝒜}\pi \left(a|S_{t+1}\right){\widehat{Q}}_n^{\pi }\left(S_{t+1},a\right)-{\widehat{Q}}_n^{\pi }\left(S_t,A_t\right),$$

and

$${\text{Δ}}^{\pi }\left(Z_t;H\right)=1-H\left(S_t,A_t\right)+\sum _{a^{\prime }}\pi \left(a^{\prime }|S_{t+1}\right)H\left(S_{t+1},a^{\prime }\right).$$

Also, the ratio estimator ${\omega }_n^{\pi }$ can be obtained from ${\widehat{H}}_n^{\pi }(\cdot ,\cdot )$ by using (4.10).

The upper optimization task (6.1) is used to search for ${\widehat{\pi }}_n$ and the two parallel lower optimization tasks (6.2)–(6.3) and (6.4)–(6.5) are used to compute two nuisance function estimators for a given π ∈ Π, i.e., the feasible set, respectively. Note that each nuisance function estimation is itself a nested optimization sub-problem. Multi-level optimization problems in general cannot be computed by iteratively updating solutions to lower problems (6.2)–(6.3) and (6.4)–(6.5), and solutions to the upper problem (6.1), in a similar manner to coordinate descent. Hence, in order to solve this problem, one common approach is to replace the inner optimization problems (6.2)–(6.3) and (6.4)–(6.5) by their corresponding Karush-Kuhn-Tucker (KKT) conditions so that the overall problem can be equivalently formulated as a nonlinear constraint optimization problem. However, this approach can be computationally expensive and may not be suitable for large scale settings. Instead we overcome this computational obstacle by using the representer theorem and obtain the closed-form solutions for our inner optimization problems (6.2)–(6.3) and (6.4)–(6.5) respectively. After plugging these closed-form solutions into (6.1), we can use a gradient-based method to find ${\widehat{\pi }}_n$.

6.1. RKHS reformulation.

In the following subsection, we briefly discuss how to simplify our multi-level optimization problem (6.1) using the representer theorem. The details of computation can be found in Appendix E. For the ease of illustration, we rewrite the training data ${𝒟}_n$ into tuples $Z_h=\left\{S_h,A_h,R_h,S_h^{\prime }\right\}$ where h = 1, . . . , N = nT indexes the tuple of the transition sample in the training set ${𝒟}_n$, S_h and $S_h^{\prime }$ are the current and next states and R_h is the associated reward. Let W_h = (S_h, A_h) be the state-action pair, and $W_h^{\prime }=\left(S_h,A_h,S_h^{\prime }\right)$. Suppose the kernel function for the state is denoted by k₀(s₁, s₂), where $s_1,s_2\in 𝒮$. In order to incorporate the action space, we can define $k\left(\left(s_1,a_1\right),\left(s_2,a_2\right)\right)={\mathbb{1}}_{\left\{a_1=a_2\right\}}{k}_0\left(s_1,s_2\right)$. Basically, we model each Q(·,a) separately for each arm in the RKHS with the same kernel k₀. Recall that we have to restrict the function space $ℱ$ such that Q(s*, a*) = 0 for all $Q\in ℱ$ so as to avoid the identification issue. Thus for any given kernel function k defined on $𝒮\times 𝒜$, we make the following transformation by defining k(W_h, W_j) = k₀(W_h, W_j) − k₀((s*, a*), W_h) − k₀((s*, a*), W_j) + k₀((s*, a*),(s*, a*)) for any 1 ≤ h, j ≤ N. One can check that the induced RKHS by k(·,·) satisfies the constraint in $ℱ$ automatically.

We denote kernel functions for $ℱ$ and $𝒢$ by k(·,·), l(·,·) respectively. The corresponding inner products are defined as ${〈\cdot ,\cdot 〉}_{ℱ}$ and ${〈\cdot ,\cdot 〉}_{𝒢}$. We first discuss the inner minimization problem (6.2)–(6.3). Note that this is indeed a nested kernel ridge regression problem, different from the standard ridge regression. The closed form solution can be obtained as ${\widehat{g}}_n^{\pi }(\cdot ,\cdot ;\eta ,Q)={\sum }_{h=1}^{N}l\left(W_h,\cdot \right)\widehat{\gamma }(\eta ,Q)$. In particular, $\widehat{\gamma }(\eta ,Q)={\left(L+\mu I_N\right)}^{-1}{\delta }_N^{\pi }(\eta ,Q)$, where $Q\in ℱ$ and L is the kernel matrix induced by l, μ = μ_nN, and ${\delta }_N^{\pi }(\eta ,Q)={\left({\delta }^{\pi }\left(Z_h;\eta ,Q\right)\right)}_{h=1}^N$ is a vector of TD error. Each TD error can be further written as ${\delta }^{\pi }\left(Z_h;\eta ,Q\right)=R-\eta -{〈Q,f_{W_h^{\prime }}〉}_{ℱ}$ where

$$f_{W_h^{\prime }}(\cdot )=k\left(W_h,\cdot \right)-\sum _{a^{\prime }}\pi \left(a^{\prime }|S_h^{\prime }\right)k\left(\left(S_h^{\prime },a^{\prime }\right),\cdot \right)\in ℱ(𝒮\times 𝒜)$$

It can be shown that ${\widehat{Q}}_n^{\pi }$ in (6.2) must be in the linear span $\{{\sum }_{h=1}^N{\alpha }_h{f}_{W_h^{\prime }}(\cdot ):{\alpha }_h\in ℝ,h=1,\dots ,N\}$ by using the representer property.

Then we can solve the optimization problem (6.2)–(6.3). The solutions for ${\{{\widehat{U}}_n^{\pi }\left(W_h^{\prime }\right)\}}_{h=1}^N$ can be found as $-\tilde{F}(\pi )\widehat{\alpha }(\pi )$ where $\tilde{F}(\pi )={({\langle f_{W_h^{\prime }},f_{W_j^{\prime }}\rangle }_{ℱ})}_{j,h=1}^N$ is a N by N matrix and $\widehat{\alpha }(\pi )$ is the vector of coefficients with a closed-form expression (see Appendix E for details). Similarly, we can compute the closed-form solutions ${\{{\widehat{g}}_n^{\pi }(W_h,{\widehat{H}}_n^{\pi })\}}_{h=1}^N$ to the problem (6.4)–(6.5) as $L\widehat{\nu }(\pi )$. Here $\widehat{\nu }(\pi )$ is the corresponding estimated coefficients associated with the kernel matrix L. The details can be found in appendix E. Note that all of these intermediate terms except for L depends on the policy π.

Summarizing together and plugging all the intermediate results into (6.1), the multi-level optimization problem can be simplified as:

$$\underset{\pi \in \text{Π}}{\text{max}}\frac{{(\widehat{\nu }(\pi ))}^{\top }L\left(R_N-\tilde{F}(\pi )\widehat{\alpha }(\pi )\right)}{\widehat{\nu }{(\pi )}^{\top }L{1}_N},$$ (6.6)

where 1_N is a length-N vector of all ones.

6.2. Optimization.

Note that problem (6.6) becomes a smooth nonlinear optimization with box constraints. We use limited-memory Broyden-Fletcher-Goldfarb-Shanno algorithm with box constraints (L-BFGS-B) to compute the solution $\widehat{\theta }$ (Liu and Nocedal, 1989). The gradient computing is provided in appendix. The computational complexity/operations of our algorithm is of order ΥN³p, where Υ is the number of iterations in our optimization algorithm. The memory requirement is of order N²p. One may implement some sub-sampling methods such as stochastic gradient decent to further improve both computation and memory complexity of our algorithm. We will leave it for future work. Although the overall optimization problem is non-convex and, thus an optimal solution may not be achievable, the performance of our numerical experiments in the following section are quite stable and promising. Recently, there is a growing interest in studying statistical properties of algorithm-type of nonconvex M-estimators, e.g., (Mei et al., 2018; Loh et al., 2017). For many practical applications, gradient decent methods with a random initialization have been demonstrated to converge to local minima (or even global minima) that are statistically good. While this is not the focus of our paper, it will be interesting to pursue toward this direction for future research such as studying the landscape of η^π and its related properties.

6.3. Tuning parameters selection.

In this subsection, we discuss the choice of tuning parameters in our method. The bandwidths in the Gaussian kernels are selected using median heuristic, e.g., median of pairwise distance (Fukumizu et al., 2009). The tuning parameters (λ_n, μ_n) and $\left({\lambda }_n^{\prime },{\mu }_n^{\prime }\right)$ are selected based on 3-fold cross-validation. Given assumptions in Theorems B.1 and B.2 of the appendix that these tuning parameters are independent of the policy π, we can select them for the ratio and value functions separately. Specifically, for the tuning parameters (λ_n, μ_n) in the estimation of value function, we focus on (6.2)–(6.3). For the tuning parameters $\left({\lambda }_n^{\prime },{\mu }_n^{\prime }\right)$ in the estimation of ratio function, we focus on (6.4)–(6.5). At the first glance, one may think the selection of tuning parameters will be the same as those in the standard supervised learning. However, this actually requires an additional step as we cannot observe responses when estimating these two coupled estimators (recalled that we need to first compute projected bellman errors), in contrast to the standard kernel regression setting. In the following, we discuss our selection procedure of (λ_n, μ_n) and $\left({\lambda }_n^{\prime },{\mu }_n^{\prime }\right)$ with more details.

Algorithm 1:

Tuning parameters selection via cross-validation

1	Input: Data ${\left\{Z_h\right\}}_{h=1}^N$, a set of M policies {π1, · · ·, πM} ⊂ Π, a set of J candidate tuning parameters ${\left\{\left({\mu }_j,{\lambda }_j\right)\right\}}_{j=1}^J$ in the value function estimation, and a set of J candidate tuning parameters ${\{({\mu }_j^{\prime },{\lambda }_j^{\prime })\}}_{j=1}^J$ in the ratio function estimation.
2	Randomly split Data into K subsets: ${\left\{Z_h\right\}}_{h=1}^N={\left\{D_k\right\}}_{k=1}^K$
3	Denote e(1) (m, j) and e(2) (m, j) as the total validation error for m-th policy and j-th pair of tuning parameters in value and ratio function estimation respectively, for m = 1, · · · M and j = 1, · · ·, J. Set their initial values as 0.
4	Repeat for m = 1, · · ·, M,
5	Repeat for k = 1, · · ·, K,
6	Repeat for j = 1, · · ·, J
7	Use ${\left\{Z_h\right\}}_{h=1}^N\backslash D_k$ to compute $\left({\widehat{\eta }}_n^{{\pi }_m},\widehat{\alpha }\left({\pi }_m\right)\right)$ and $\widehat{\nu }({\pi }_m)$ by (6.2)–(6.3) and (6.4)–(6.5) using tuning parameters (μj, λj) and $({\mu }_j^{\prime },{\lambda }_j^{\prime })$ respectively;
8	Compute ${\delta }^{{\pi }_m}(\cdot ;\widehat{\eta }\left({\pi }_m\right),{\widehat{Q}}_n^{{\pi }_m})$ and ${\epsilon }^{{\pi }_m}(\cdot ;{\widehat{H}}_n^{{\pi }_m})$ and their corresponding squared Bellman errors mse(1) and mse(2) on the dataset Dk by Gaussian kernel regression;
9	Assign e(1) (m, j) = e(1) (m, j) + mse(1) and e(2) (m, j) = e(2)(m, j) + mse(2);
10	Compute j(1)* ∈ argminj maxm e(1) (m, j) and j(2)* ∈ argminj maxm e(2) (m, j)
11	Output: $({\mu }_{j^{(1)\ast }}^{(1)},{\lambda }_{j^{(1)\ast }}^{(1)})$ and $({\mu }_{j^{(2)\ast }}^{\prime },{\lambda }_{j^{(2)\ast }}^{\prime })$.

We first randomly choose a set of candidate policies used to gauge our tuning parameters. For each candidate policy, π, in this set, we can firstly estimate $\left({\widehat{\eta }}_n^{\pi },\widehat{\alpha }(\pi )\right)$ by the proposed method using two folds of data. Then for the value function estimation, we calculate temporal difference errors ${\delta }^{\pi }\left(\cdot ;{\widehat{\eta }}_n^{\pi },\widehat{\alpha }(\pi )\right)$ for each transition sample in the validation set. Since we cannot observe/calculate the true bellman error, following the idea in (Farahmand and Szepesvári, 2011), we estimate the Bellman error by projecting these temporal differences on the space of $𝒮\times A$ in the validation set using the standard Gaussian kernel regression. Thus for each policy π and each pair of tuning parameters, we output the squared estimated Bellman error in the validation set as a criterion to evaluate the performance of our value function estimation. Since tuning parameters are assumed independent of policies, we then select the tuning parameters that minimize the worst case of estimated Bellman errors among the set of all candidate policies. We use the same strategy to select the tuning parameters for our ratio estimation. The details are given in the Algorithm 1. Without the independent assumptions of tuning parameters from the policies in Π, one may alternatively choose these tuning parameters jointly by maximizing ${\widehat{\eta }}_n^{\pi }$ on the validation set, which requires large computational costs and we omit here. But it would be very interesting to study the theoretical properties of these two cross-validation procedures, or more generally, the selection of tuning parameters in the framework of couple estimation, which we leave it as future work.

7. Simulation Studies.

In this section, we consider two scenarios to evaluate the proposed algorithm. For both scenarios, we consider S_t = (S_t,1, S_t,2, S_t,3) as a three-dimensional state at each decision point t, and the action space is binary, i.e., $𝒜=\{0,1\}$. The behavior policy used to generate actions follows Bernoulli distribution with equal probabilities. In addition, the initial state S₁ is sampled from standard multi-variate normal distribution, i.e., S₁ ∼ MV N(0, I₄)

The first scenario we consider is a standard MDP setting. Let ξ_t follows a standard multivariate normal distribution. Then we generate the transition of states and reward functions via following models:

$$S_{t+1,1}=0.5S_{t,1}+2{\xi }_{t,1},$$

$$S_{t+1,2}=0.25S_{t,2}+0.125A_t+2{\xi }_{t,2},$$

$$S_{t+1,3}=0.9S_{t,3}+0.05S_{t,3}{A}_t+0.5A_t+{\xi }_{t,3},$$

$$R_{t+1}=10-0.4S_{t,3}+0.25S_{t,1}{A}_t\times \left(0.04+0.02S_{t,1}+0.02S_{t,2}\right)+0.16{\xi }_{t,4},$$

for t = 1, · · ·, T. Here S_t,3 can be interpreted as the treatment burden or fatigue.

The second scenario we consider is a non-stationary environment. In particular, we consider the same transition models as above, but let the reward function to be time-dependent. More specifically, we consider

$$R_{t+1}=10-{\tau }_t{S}_{t,3}+{\beta }_t{S}_{t,1}{A}_t\left(0.04+0.02S_{t,1}+0.02S_{t,2}\right),$$

where the time-varying parameters β_t = 0.25×exp(−0.05(t−1)) and τ_t = 0.4×exp(−0.05(t−1)). This generative model represents the scenario in which as the study progresses, the overall impact of intervention is decreasing. Note that since the reward function is non-stationary, we do not have a guarantee for our proposed algorithm to find an optimal policy.

We compare with four baseline methods, which were proposed in the setting of the discounted sum of rewards. The first two are recently proposed deep off-policy RL algorithms (Fujimoto, Meger and Precup, 2019; Kumar et al., 2019) denoted by BCQ and BEAR respectively. The underlying idea behind these two state-of-art algorithms is to conservatively estimate the optimal Q-function on the less explored state-action pair and restrict the resulting policy close to the behavior one. The third method is the celebrated fitted-Q iteration (FQI) method proposed by Ernst et al. (2005). At each iteration, relying on the optimal Bellman equation, FQI algorithm updates the estimation of the optimal Q-function via solving a supervised learning problem. The last method is V-learning proposed by (Luckett et al., 2019), which also aims to learn an optimal in-class policy. Since our goal is to maximize the long-term average reward, we set the discount factor γ in these four methods as 0.99 to approximate the average reward for comparison. In addition, to draw a relatively fair comparison, we implement these four methods using the same policy class as ours. Specifically, the first three methods will output an estimation of the optimal Q-function (defined in the discounted setting), after which we implement a weighted logistic regression to estimate the optimal in-class policy. For V-learning, we keep the default setup and use the same policy class as ours. Finally, for BEAR and BCQ, we use two-hidden layers neural networks with 32 nodes for each and ReLU activation functions to model the optimal Q-function. The other hyper-parameters are either tuned for their best performance, or recommended in the official implementation as robust choices. For FQI, we implement a kernel ridge regression at each iteration with tuning parameters selected similar to our nuisance parameter estimation.

To demonstrate the performance of our algorithm compared with other methods, we consider different combinations of the number of trajectories n and the length of each trajectory T. Specifically, we consider (n, T) = (40, 50), (40, 100), and (80, 50). Once all estimated policies are obtained, we generate another 100 test samples with the length of trajectories 1000 using all learned policies and compute the corresponding empirical average of observed rewards. In order to compare results with the best in-class stationary policy, we combine the gradient-type optimization algorithm with Monte Carlo method to estimate the best in-class policy that can maximize ${\mathbb{E}}^{\pi }\left[\frac{1}{1000}{\sum }_{t=1}^{1000}{R}_{t+1}\right]$. Specifically, for each policy with parameter θ, we generate a sample of n = 100 and T = 1000 to approximate ${\mathbb{E}}^{\pi }\left[\frac{1}{1000}{\sum }_{t=1}^{1000}{R}_{t+1}\right]$ by the empirical average of rewards. Then we apply L-BFGS algorithm and require θ to be between −10 to 10 to search for the best in-class stationary policy, which is treated as the oracle policy.

Results of the above two scenarios can be found in Table 1. As we can see, our algorithm performs well in finding optimal in-class stationary policies, compared with the other four baseline methods. Compared with the oracle one with the best average reward about 10, the regret of our algorithm is almost the smallest among all these methods, which is expected as we aim to maximize the average reward while the other four methods are for maximizing the discounted sum of rewards. For BEAR and BCQ with neural network models, due to the relatively small sample size and a large discount factor γ, the performances seem unstable. FQI and V-learning methods overall show competitive performances. But it can be seen that V-learning may suffer some large variance. In addition, one possible reason for the high quality performance of our method in Scenario 2 is that the time-dependent effect in reward function is exponentially decaying by our design. Therefore we expect the performance of our algorithm may not be affected severely by the non-stationarity. In addition, it can be seen that as the sample size n or the length of each trajectory T increases, the average rewards of our estimated policies are also improved, demonstrating the appealing performance of the proposed method. Finally, we remark that the maximum running time of our method for one replication in our simulation studies is less than 40 minutes.

Table 1.

Monte Carlo estimation of the average rewards of the learned policy from proposed algorithm and three baseline offline RL algorithms over T = 1000 with 100 replications. Numbers in parentheses are corresponding standard deviations. The oracle in-class optimal average rewards for both scenario are about 10.002.

	n	T	Our method	BEAR	BCQ	FQI	V-learning
Scenario 1	40	50	9.215 (0.133)	7.513 (0.044)	8.187 (0.067)	9.728 (0.007)	9.246 (0.470)
	40	100	9.913 (0.050)	6.949 (0.053)	7.362 (0.068)	9.820 (0.028)	9.345 (0.476)
	80	50	9.834 (0.052)	7.487 (0.033)	7.992 (0.059)	9.764 (0.005)	9.461 (0.457)
Scenario 2	40	50	9.243 (0.133)	9.128 (0.009)	9.426 (0.020)	9.579 (0.028)	9.834 (0.097)
	40	100	9.905 (0.006)	9.508 (0.006)	9.692 (0.012)	9.858 (0.011)	9.840 (0.108)
	80	50	9.919 (0.005)	9.141 (0.011)	9.384 (0.020)	9.652 (0.025)	9.873 (0.097)

8. Application to mobile health.

We apply the proposed method to HeartSteps. HeartSteps is mobile health application focusing on physical activity. Three studies were conducted to develop the intervention. In this work, we apply the proposed method to the data collected from the first study, which we will refer to as HS1 in the throughout. HS1 is a 42-day micro-randomized trial (Klasnja et al. (2015); Liao et al. (2016)). Each participant was provided with a Jawbone wrist tracker to collect step count data and specified five decision times, roughly 2.5 hours apart during each day, that would be good times to potentially receive contextually tailored activity suggestion message. In HS1, the activity message was sent with a fixed probability 0.6 at each of the five decision times. Our goal is to use HS1 data to learn a treatment policy that determines at each decision time whether to send the activity message (i.e., binary action).

We construct the state variable using the previous step count (the 30-min step count prior to the decision time and from yesterday), location, temperature and past notifications. We set the reward to be the log transformation of the step count in 30-min window after each decision time. In this analysis, we include 37 participants’ data and exclude the decision times when participants were traveling abroad or experiencing technical issues or when the reward (i.e., post 30-min step count) was considered as missing (Klasnja et al., 2018).

Next, we construct the policy class. In this analysis, we include two state variables in the policy. The first variable is the location (home/work vs. other locations). Location is important because people, in a more structured environment (i.e. at home or work), may respond better to an activity suggestion as compared to when they are at other locations. As a proxy for participant burden, the second variable included in the policy is “dosage”, a discounted sum of the number of past activity messages sent with the discount rate chosen as 0.95. The rationale for using this variable is that receiving too many notifications in the recent past is likely to decrease the effectiveness of sending the activity message due to over-burdening participants. We consider the policy class of the form π_θ(1|s) = expit(θ^⊤ϕ(s)), θ ∈ Θ, where the feature vector ϕ(s) = (1, dosage, location) and Θ is the box constraint within −10 and 10. Here dosage is standardized to be within 0 and 1.

We apply the proposed method with the tuning parameter selected by cross-validation in Algorithm 1. The estimated coefficients are [10, −10, −4.788]. Figure 1 shows the estimated policy at different combination of dosage and location. As one would expect, the learned policy tends to send fewer suggestions if the participant received many suggestions in the recent past. Also, the policy indicates that it is more effective to send the message when the user is at home/work location. The estimated average reward of this policy is 3.301. As a comparison, the estimated average reward of the simple location-based policy (i.e., send only when the user is at home/work) is 3.15 and the send-nothing policy is 2.96. Transforming to the scale of the raw step count as that in (Klasnja et al., 2018), the learned policy can result in 16% (i.e., exp(3.301 − 3.15) − 1 = 0.16) improvement, which is equivalent to 40 more steps (the mean step count across all decision times in the data is 248) compared with the simple location-based policy, and 40% (i.e., exp(3.301 − 2.96) − 1 = 0.40) improvement, or equivalently 101 steps more, compared with the send-nothing policy. Lastly, we remark that the running time of our real data analysis is about 2 hours, which is acceptable in the batch setting. This is ultimately different from online RL domains where the policy is usually updated upon the arrival of each observation.

Fig 1.

The estimated policy in HeartSteps data.

9. Discussion.

Double/Debiased machine learning

An alternative way to construct the estimator for the average reward is based on the idea of double/debiased machine learning (a.k.a. cross-fitting, Bickel et al. (1993) and Chernozhukov et al. (2018)). There is growing interest in using double machine learning in causal inference and in the policy learning literature (Zhao et al., 2019) in order to relax assumptions on the convergence rates of nuisance parameters. The basic idea is to split the data into K folds. For each of the K folds, construct the estimating equation by plugging in the estimated nuisance functions that are obtained using the remaining (K − 1) folds. The final estimator is obtained by solving the aggregated estimation equations. While cross-fitting requires weaker conditions on the nuisance function estimations, it indeed incurs additional computational cost, especially in our setting where nuisance functions are policy-dependent and we aim to search for the in-class optimal policy. Further, this sample splitting procedure may not be stable when the sample size is relatively small, e.g., in a typical mHealth clinical trial. A more efficient way of data splitting under the framework of MDP is needed, which we leave as future work

Computation and optimization

Our current algorithm requires relatively large computation and memory because of the non-parametric estimation and the policy-dependent structure of nuisance functions. It is therefore desirable to develop a more efficient algorithm. One possible remedy is to consider a zero-order optimization method such as Bayesian optimization (Snoek, Larochelle and Adams, 2012), which is suitable when the dimension of state variables is small. Another possible way to improve the computational efficiency is to first apply some simple algorithm to estimate a sub-optimal policy, based on which we can implement our method to estimate two nuisance parameters. Then one can develop the performance difference lemma in terms of the average reward MDP, similar to that in the discounted setting (Kakade and Langford, 2002), to construct a lower bound for $𝒱(\pi )$ using two estimated nuisance parameters. The last step is to optimize this lower bound for obtaining a better policy. This method may require less computational cost.

Tuning parameters/Model selection

In our proposed algorithm, we assume tuning parameters are independent of policies, based on which we develop a min-max cross-validation procedure for the selection of tuning parameters. Model selection in the offline RL setting, which is necessary for improving generalization of RL techniques, is often considered as a challenging task as there is no ground truth available for performance demonstration, in contrast to the online setting with simulated environment. Therefore, it will be interesting to systematically investigate how to perform model selection in offline RL and to provide theoretical guarantees.