Sample-Efficient Reinforcement Learning for Linearly-Parameterized MDPs with a Generative Model

Abstract

The curse of dimensionality is a widely known issue in reinforcement learning (RL). In the tabular setting where the state space $\mathcal{S}$ and the action space $\mathcal{A}$ are both finite, to obtain a nearly optimal policy with sampling access to a generative model, the minimax optimal sample complexity scales linearly with $|\mathcal{S}|\times |\mathcal{A}|$, which can be prohibitively large when $\mathcal{S}$ or $\mathcal{A}$ is large. This paper considers a Markov decision process (MDP) that admits a set of state-action features, which can linearly express (or approximate) its probability transition kernel. We show that a model-based approach (resp. Q-learning) provably learns an ε-optimal policy (resp. Q-function) with high probability as soon as the sample size exceeds the order of $\frac{K}{{(1-\gamma )}^3{\epsilon }^2}\left(\text{resp}.\frac{K}{{(1-\gamma )}^4{\epsilon }^2}\right)$, up to some logarithmic factor. Here K is the feature dimension and γ ∈ (0, 1) is the discount factor of the MDP. Both sample complexity bounds are provably tight, and our result for the model-based approach matches the minimax lower bound. Our results show that for arbitrarily large-scale MDP, both the model-based approach and Q-learning are sample-efficient when K is relatively small, and hence the title of this paper.

1. Introduction

Reinforcement learning (RL) studies the problem of learning and decision making in a Markov decision process (MDP). Recent years have seen exciting progress in applications of RL in real world decision-making problems such as AlphaGo [SHM+16, SSS+17] and autonomous driving [KST+21]. Specifically, the goal of RL is to search for an optimal policy that maximizes the cumulative reward, based on sequential noisy data. There are two popular approaches to RL: model-based and model-free ones.

• The model-based approaches start with formulating an empirical MDP by learning the probability transition model from the collected data samples, and then estimating the optimal policy / value function based on the empirical MDP.

• The model-free approaches (e.g. Q-learning) learn the optimal policy or the optimal (action-)value function from samples. As its name suggests, model-free approaches do not attempt to learn the model explicitly.

Generally speaking, model-based approaches enjoy great flexibility since after the transition model is learned in the first place, it can then be applied to any other problems without touching the raw data samples. In comparison, model-free methods, due to its online nature, are usually memory-efficient and can interact with the environment and update the estimate on the fly.

This paper is devoted to investigating the sample efficiency of both model-based RL and Q-learning (arguably one of the most commonly adopted model-free RL algorithms). It is well known that MDPs suffer from the curse of dimensionality. For example, in the tabular setting where the state space $\mathcal{S}$ and the action space $\mathcal{A}$ are both finite, to obtain a near optimal policy or value function given sampling access to a generative model, the minimax optimal sample complexity scales linearly with $|\mathcal{S}|\times |\mathcal{A}|$ [AMK13, AKY20]. However contemporary applications of RL often encounters environments with exceedingly large state and action spaces, whilst the data collection might be expensive or even high-stake. This suggests a large gap between the theoretical findings and practical decision-making problems where $|\mathcal{S}|$ and $|\mathcal{A}|$ are large or even infinite.

To close the aforementioned theory-practice gap, one natural idea is to impose certain structural assumption on the MDP. In this paper we follow the feature-based linear transition model studied in [YW19], where each state-action pair $(s,a)\in \mathcal{S}\times \mathcal{A}$ admits a K dimensional feature vector $\varphi (s,a)\in {ℝ}^K$ that expresses the transition dynamics $\mathbb{P}(\cdot \mid s,a)=\Psi \varphi (s,a)$ for some unknown matrix $\Psi \in {ℝ}^{|\mathcal{S}|\times K}$ which is common for all (s, a). This model encompasses both the tabular case and the homogeneous model in which the state space can be partitioned into K equivalent classes. Assuming access to a generative model [Kak03, KS99], under this structural assumption, this paper aims to answer the following two questions:

How many samples are needed for model-based RL and Q-learning to learn an optimal policy under the feature-based linear transition model?

In what follows, we will show that the answer to this question scales linearly with the dimension of the feature space K and is independent of $|\mathcal{S}|$ and $|\mathcal{A}|$ under the feature-based linear transition model. With the aid of this structural assumption, model-based RL and Q-learning becomes significantly more sample-efficient than that in the tabular setting.

Our contributions.

We focus our attention on an infinite horizon MDP with discount factor γ ∈ (0, 1). We use ε-optimal policy to indicate the policy whose expected discounted cumulative rewards are ε close to the optimal value of the MDP. Our contributions are two-fold:

• We demonstrate that model-based RL provably learns an ε-optimal policy by performing planning based on an empirical MDP constructed from a total number of

  $$\tilde{O}\left(\frac{K}{{(1-\gamma )}^3{\epsilon }^2}\right)$$

  samples, for all ε ∈ (0, (1 − γ)^−1/2]. Here $\tilde{O}(\cdot )$ hides logarithmic factors compared to the usual O(·) notation. To the best of our knowledge, this is the first theoretical guarantee for model-based RL under the feature-based linear transition model. This sample complexity bound matches the minimax limit established in [YW19] up to logarithmic factor.
• We also show that Q-learning provably finds an entrywise ε-optimal Q-function using a total number of

  $$\tilde{O}\left(\frac{K}{{(1-\gamma )}^4{\epsilon }^2}\right)$$

  samples, for all ε ∈ (0, 1]. This sample complexity upper bound improves the state-of-the-art result in [YW19] and the dependency on the effective horizon (1 − γ)⁻⁴ is sharp in view of [LCC+21a].

These results taken collectively show the minimax optimality of model-based RL and the sub-optimality of Q-learning in sample complexity.

2. Problem formulation

This paper focuses on tabular MDPs in the discounted infinite-horizon setting [B+00]. Here and throughout, ${\Delta }_{d-1}≔\left\{v\in {ℝ}^d:\sum _{i=1}^d{v}_i=1,v_i\ge 0,\forall i\in [d]\right\}$ stands for the d-dimensional probability simplex and [N] ≔ {1, 2, ⋯, N} for any $N\in {\mathbb{N}}^{+}$.

Discounted infinite-horizon MDPs.

Denote a discounted infinite-horizon MDP by a tuple $M=(\mathcal{S},\mathcal{A},P,r,\gamma )$, where $\mathcal{S}=\{1,\cdots ,|\mathcal{S}|\}$ is a finite set of states, $\mathcal{A}=\{1,\cdots ,|\mathcal{A}|\}$ is a finite set of actions, $P : \mathcal{S}\times \mathcal{A}\to {\Delta }_{|\mathcal{S}|-1}$ represents the probability transition kernel where P(s′|s, a) denotes the probability of transiting from state s to state s′ when action a is taken, $r : \mathcal{S}\times \mathcal{A}\to [0,1]$ denotes the reward function where r(s, a) is the instantaneous reward received when taking action $a\in \mathcal{A}$ while in state $s\in \mathcal{S}$, and γ ∈ (0, 1) is the discount factor.

Value function and Q-function.

Recall that the goal of RL is to learn a policy that maximizes the cumulative reward, which corresponds to value functions or Q-functions in the corresponding MDP. For a deterministic policy $\pi :\mathcal{S}\to \mathcal{A}$ and a starting state $s\in \mathcal{S}$, we define the value function as

$$V^{\pi }(s)≔\mathbb{E}\left[\sum _{k=0}^{\infty }{\gamma }^{k}r\left(s_k,a_k\right)\mid s_0=s\right]$$

for all $s\in \mathcal{S}$. Here, the trajectory is generated by a_k = π(s_k) and s_k+1 ~ P(s_k+1|s_k, a_k) for every k ≥ 0. This function measures the expected discounted cumulative reward received on the trajectory {(s_k, a_k)}_k≥0 and the expectation is taken with respect to the randomness of the transitions s_k+1 ~ P(·|s_k, a_k) on the trajectory. Recall that the immediate rewards lie in [0, 1], it is easy to derive that $0\le V^{\pi }(s)\le \frac{1}{1-\gamma }$ for any policy π and state s. Accordingly, we define the Q-function for policy π as

$$Q^{\pi }(s,a)≔\mathbb{E}\left[\sum _{k=0}^{\infty }{\gamma }^{k}r\left(s_k,a_k\right)\mid s_0=s,a_0=a\right]$$

for all $(s,a)\in \mathcal{S}\times \mathcal{A}$. Here, the actions are chosen by the policy π except for the initial state (i.e. a_k = π(s_k) for all k ≥ 1). Similar to the value function, we can easily check that $0\le Q^{\pi }(s,a)\le \frac{1}{1-\gamma }$ for any π and (s, a). To maximize the value function or Q function, previous literature [BD59,SB18] establishes that there exists an optimal policy π⋆ which simultaneously maximizes V^π(s) (resp. Q^π(s, a)) for all $s\in \mathcal{S}\text{ (resp. }(s,a)\in \mathcal{S}\times \mathcal{A})$. We define the optimal value function V⋆ and optimal Q-function Q⋆ respectively as

$$V^{\star }(s)≔\underset{\pi }{\text{max}}{V}^{\pi }(s)=V^{{\pi }^{\star }}(s),\hspace{1em}{Q}^{\star }(s,a)≔\underset{\pi }{\text{max}}{Q}^{\pi }(s,a)=Q^{{\pi }^{\star }}(s,a)$$

for any state-action pair $(s,a)\in \mathcal{S}\times \mathcal{A}$.

Linear transition model.

Given a set of K feature functions ${\varphi }_1,{\varphi }_2,\cdots ,{\varphi }_K:\mathcal{S}\times \mathcal{A}\to ℝ$, we define ϕ to be a feature mapping from $\mathcal{S}\times \mathcal{A}$ to ${ℝ}^K$ such that

$$\varphi (s,a)=\left[{\varphi }_1(s,a),\cdots ,{\varphi }_K(s,a)\right]\in {ℝ}^K.$$

Then we are ready to define the linear transition model [YW19] as follows.

Definition 1 (Linear transition model).

Given a discounted infinite-horizon MDP $M=(\mathcal{S},\mathcal{A},P,r,\gamma )$ and a feature mapping $\varphi :\mathcal{S}\times \mathcal{A}\to {ℝ}^K$, M admits the linear transition model if there exists some (unknown) functions ${\psi }_1,\cdots ,{\psi }_K:\mathcal{S}\to ℝ$, such that

$$P\left(s^{\prime }\mid s,a\right)=\sum _{k=1}^K{\varphi }_k(s,a){\psi }_k\left(s^{\prime }\right)$$ (2.1)

for every $(s,a)\in \mathcal{S}\times \mathcal{A}$ and $s^{\prime }\in \mathcal{S}$.

Readers familiar with linear MDP literatures might immediately recognize that the above definition is the same as the structure imposed on the probability transition kernel P in the linear MDP model [YW19,JYWJ20,ZBB+20,HZG21,TV20,WDYS20,WJLJ21]. However unlike linear MDP which also requires the reward function r(s, a) to be linear in the feature mapping ϕ(s, a), here we do not impose any structural assumption on the reward.

Example 1 (Tabular MDP).

Each tabular MDP can be viewed as a linear transition model with feature mapping $\varphi (s,a)=e_{(s,a)}\in {ℝ}^{|\mathcal{S}|\times |\mathcal{A}|}$ (i.e. the vector with all entries equal to 0 but the one corresponding to (s, a) equals to 1) for all $(s,a)\in \mathcal{S}\times \mathcal{A}$. To see this, we can check that Definition 1 is satisfied with $K=|\mathcal{S}|\times |\mathcal{A}|$ and ${\psi }_{(s,a)}\left(s^{\prime }\right)=\mathbb{P}\left(s^{\prime }\mid s,a\right)$ for each s, $s^{\prime }\in \mathcal{S}$ and $a\in \mathcal{A}$. This example is a sanity check of Definition 1, which also shows that our results (Theorem 1 and 2) can recover previous results on tabular MDP [AKY20,LCC+21a] by taking $K=|\mathcal{S}|\times |\mathcal{A}|$.

Example 2 (Simplex Feature Space).

If all feature vectors ${\{\varphi (s,a)\}}_{(s,a)\in \mathcal{S}\times \mathcal{A}}$ fall in the probability simplex Δ_K−1, a linear transition model can be constructed by taking ψ_k(·) to be any probability measure over $\mathcal{S}$ for all k ∈ [K].

A key observation is that the model size of linear transition model with known feature mapping ϕ is $|\mathcal{S}|K$ (the number of coefficients ψ_k (s′) in (2.1)), which is still large when the state space $\mathcal{S}$ is large. In contrast, it will be established later that to learn a near-optimal policy or Q-function, we only need a much smaller number of samples, which depends linearly on K and is independent of $|\mathcal{S}|$.

Next, we introduce a critical assumption employed in prior literature [YW19,ZLKB19,SS20].

Assumption 1 (Anchor state-action pairs).

Assume there exists a set of anchor state-action pairs $\mathcal{K}\subset \mathcal{S}\times \mathcal{A}$ with $|\mathcal{K}|=K$ ¹ such that for any $(s,a)\in \mathcal{S}\times \mathcal{A}$, its corresponding feature vector can be expressed as a convex combination of the feature vectors of anchor state-action pairs $\{(s,a) : (s,a)\in \mathcal{K}\}$:

$$\varphi (s,a)=\sum _{i:\left(s_i,a_i\right)\in \mathcal{K}}{\lambda }_i(s,a)\varphi \left(s_i,a_i\right)\hspace{1em}\mathit{\text{for}}\hspace{1em}\sum _{i=1}^K{\lambda }_i(s,a)=1\hspace{1em}\mathit{\text{and}}\hspace{1em}{\lambda }_i(s,a)\ge 0.$$ (2.2)

Further, we assume that the vectors in $\{\varphi (s,a):(s,a)\in \mathcal{K}\}$ are linearly independent.

We pause to develop some intuition of this assumption using Examples 1 and 2. In Example 1, it is straightforward to check that tabular MDPs satisfies Assumption 1 with $\mathcal{K}=\mathcal{S}\times \mathcal{A}$. In terms of Example 2, without loss of generality we can assume that the subspace spanned by the features has full rank, i.e. $\text{span}\{\varphi (s,a) : (s,a)\in \mathcal{S}\times \mathcal{A}\}={ℝ}^K$ (otherwise we can reduce the dimension of feature space). Then we can also check that Example 2 satisfies Assumption 1 with arbitrary $\mathcal{K}\subseteq \mathcal{S}\times \mathcal{A}$ such that the vectors in $\{\varphi (s,a):(s,a)\in \mathcal{K}\}$ are linearly independent. In fact, this sort of “anchor” notion appears widely in the literature: [AGM12] considers “anchor word” in topic modeling; [DS04] defines “separability” in their study of non-negative matrix factorization; [SJJ95] introduces “aggregate” in reinforcement learning; [DKW19] studies “anchor state” in soft state aggregation models. These concepts all bear some kind of resemblance to our definition of anchor state-action pairs here.

Throughout this paper, we assume that the feature mapping ϕ is known, which is a widely adopted assumption in previous literature [YW19,JYWJ20,ZHG21,HZG21,TV20,WDYS20,WJLJ21]. In practice, large scale RL usually makes use of representation learning to obtain the feature mapping ϕ. Furthermore, the learned representations can be selected to satisfy the anchor state-action pairs assumption by design.

A useful implication of Assumption 1 is that we can represent the transition kernel as

$$P(\cdot \mid s,a)=\sum _{i:\left(s_i,a_i\right)\in \mathcal{K}}{\lambda }_i(s,a)P\left(\cdot \mid s_i,a_i\right),$$ (2.3)

This follows simply from substituting (2.2) into (2.1) (see (A.4) in Appendix A for a formal proof).

3. Model-based RL with a generative model

We start with studying model-based RL with a generative model in this section. We propose a model-based planning algorithm and show that it returns an ε-optimal policy with minimax optimal sample size.

3.1. Main results

A generative model and an empirical MDP.

We assume access to a generative model that provides us with independent samples from M. For each anchor state-action pair $\left(s_i,a_i\right)\in \mathcal{K}$, we collect N independent samples $s_i^{(j)}~P\left(\cdot \mid s_i,a_i\right),j\in [N]$. This allows us to construct an empirical transition kernel $\widehat{P}$ where

$$\widehat{P}\left(s^{\prime }\mid s,a\right)=\sum _{i=1}^K{\lambda }_i(s,a)\cdot \left(\frac{1}{N}\sum _{j=1}^N\mathbb{1}\left\{s_i^{(j)}=s^{\prime }\right\}\right),$$ (3.1)

for each $(s,a)\in \mathcal{S}\times \mathcal{A}$. Here, $\frac{1}{N}\sum _{j=1}^N\mathbb{1}\left\{s_i^{(j)}=s^{\prime }\right\}$ is an empirical estimate of P(s′|s_i, a_i) and then (2.3) is employed. With $\widehat{P}$ in hand, we can construct an empirical MDP $\widehat{M}=(\mathcal{S},\mathcal{A},\widehat{P},r,\gamma )$. Our goal here is to derive the sample complexity which guarantees that the optimal policy of $\widehat{M}$ is an ε-optimal policy for the true MDP M. The algorithm is summarized below.

Careful readers may note that in Algorithm 1, $\{\lambda (s,a):(s,a)\in \mathcal{S}\times \mathcal{A}\}$ is used in the construction of $\widehat{P}$, while $\{\lambda (s,a):(s,a)\in \mathcal{S}\times \mathcal{A}\}$ is not input into the algorithm. This is because given $\mathcal{K}$ and ϕ are known, $\{\lambda (s,a):(s,a)\in \mathcal{S}\times \mathcal{A}\}$ can be calculated explicitly. The following theorem provides theoretical guarantees for the output policy $\widehat{\pi }$ of the chosen optimization algorithm on the empirical MDP $\widehat{M}$.

Theorem 1.

Suppose that δ > 0 and ε ∈ (0, (1 − γ)^−1/2]. Let $\widehat{\pi }$ be the policy returned by Algorithm 1. Assume that

$$N\ge \frac{C\text{log}(K/((1-\gamma )\delta ))}{{(1-\gamma )}^3{\epsilon }^2}$$ (3.2)

for some sufficiently large constant C > 0. Then with probability exceeding 1 − δ,

$$Q^{\star }(s,a)-Q^{\widehat{\pi }}(s,a)\le \epsilon +\frac{4{\epsilon }_{\text{opt}}}{1-\gamma },$$ (3.3)

for every $(s,a)\in \mathcal{S}\times \mathcal{A}$. Here ε_opt is the target algorithmic error level in Algorithm 1.

We first remark that the two terms on the right hand side of (3.3) can be viewed as statistical error and algorithmic error, respectively. The first term ε denotes the statistical error coming from the deviation of the empirical MDP $\widehat{M}$ from the true MDP M. As the sample size N grows, ε could decrease towards 0. The other term 4ε_opt/(1 − γ) represents the algorithmic error where ε_opt is the target accuracy level of the planning algorithm applied to $\widehat{M}$. Note that ε_opt can be arbitrarily small if we run the planning algorithm (e.g. value iteration) for enough iterations. A few implications of this theorem are in order.

• Minimax-optimal sample complexity. Assume that ε_opt is made negligibly small, e.g. ε_opt = O((1 − γ)ε) to be discussed in the next point. Note that we draw N independent samples for each state-action pair $(s,a)\in \mathcal{K}$, therefore the requirement (3.2) for finding an O(ε)-optimal policy translates into the following sample complexity requirement

  $$\tilde{O}\left(\frac{K}{{(1-\gamma )}^3{\epsilon }^2}\right).$$

  This matches the minimax optimal lower bound (up to a logarithm factor) established in [YW19, Theorem 1] for feature-based MDP. In comparison, for tabular MDP the minimax optimal sample complexity is $\tilde{\Omega }\left({(1-\gamma )}^{-3}{\epsilon }^{-2}|\mathcal{S}\parallel \mathcal{A}|\right)$ [AMK13,AKY20]. Our sample complexity scales linearly with K instead of $|\mathcal{S}||\mathcal{A}|$ for tabular MDP as desired.
• Computational complexity. An advantage of Theorem 1 is that it incorporates the use of any efficient planning algorithm applied to the empirical MDP $\widehat{M}$. Classical algorithms include Q-value iteration (QVI) or policy iteration (PI) [Put14]. For example, QVI achieves the target level ε_opt in $O\left({(1-\gamma )}^{-1}\text{log}{\epsilon }_{\text{opt}}^{-1}\right)$ iterations, and each iteration takes time proportional to $O(NK+|\mathcal{S}\parallel \mathcal{A}|K)$. To learn an O(ε)-optimal policy, which requires sample complexity (3.2) and the target level ε_opt = O((1 − γ)ε), the overall running time is

  $$\tilde{O}\left(\frac{\left|\mathcal{S}\right|\left|\mathcal{A}\right|K}{1-\gamma }+\frac{K}{{(1-\gamma )}^4{\epsilon }^2}\right).$$

  In comparison, for the tabular MDP the corresponding running time is $\tilde{O}\left({(1-\gamma )}^{-4}{\epsilon }^{-2}|\mathcal{S}||\mathcal{A}|\right)$ [AKY20]. This suggests that under the feature-based linear transition model, the computational complexity is $\text{min}\left\{|\mathcal{S}||\mathcal{A}|/K,{(1-\gamma )}^{-3}{\epsilon }^{-2}/K\right\}$ times lower than that for the tabular MDP (up to logarithm factors), which is significantly more efficient when K is not too large.
• Stability vis-à-vis model misspecification. A more general version of Theorem 1 (Theorem 3 in Appendix B) shows that when P approximately (instead of exactly) admits the linear transition model, we can still achieve some meaningful result. Specifically, if there exists a linear transition kernel $\tilde{P}$ obeying ${\text{max}}_{(s,a)\in \mathcal{S}\times \mathcal{A}}\parallel \tilde{P}(\cdot \mid s,a)-P(\cdot \mid s,a){\parallel }_1\le \xi$ for some ξ ≥ 0, we can show that $\widehat{\pi }$ returned by Algorithm 1 (with slight modification) satisfies

  $$Q^{\star }(s,a)-Q^{\widehat{\pi }}(s,a)\le \epsilon +\frac{4{\epsilon }_{\text{opt}}}{1-\gamma }+\frac{22\xi }{{(1-\gamma )}^2},$$

  for every $(s,a)\in \mathcal{S}\times \mathcal{A}$. This shows that the model-based method is stable vis-á-vis model misspecification. Interested readers are referred to Appendix B for more details.

In Algorithm 1, the reward function r is assumed to be known. If the information of r is unavailable, an alternative is to assume that r is linear with respect to the feature mapping ϕ, i.e. r(s, a) = θ^⊤ϕ(s, a) for every $(s,a)\in \mathcal{S}\times \mathcal{A}$, which is widely adopted in linear MDP literature [HZG21,JYWJ20,WDYS20,WJLJ21]. Under this linear assumption, one can obtain θ by solving the following linear system of equations

$$r(s,a)={\theta }^{\top }\varphi (s,a),\forall (s,a)\in \mathcal{K},$$ (3.4)

which can be constructed by the observed reward r(s, a) for all anchor state-action pairs.

4. Model-free RL—vanilla Q Learning

In this section, we turn to study one of the most popular model-free RL algorithms—Q-learning. We provide tight sample complexity bound for vanilla Q-learning under the feature-based linear transition model, which shows its sample-efficiency (depends on |K| instead of $|\mathcal{S}|$ or $|\mathcal{A}|$) and sub-optimality in the dependency on the effective horizon.

4.1. Q-learning algorithm

The vanilla Q-learning algorithm maintains a Q-function estimate $Q_t:\mathcal{S}\times \mathcal{A}\to ℝ$ for all t ≥ 0, with initialization Q₀ obeying $0\le Q_0(s,a)\le \frac{1}{1-\gamma }$ for every $(s,a)\in \mathcal{S}\times \mathcal{A}$. Assume we have access to a generative model. In each iteration t ≥ 1, we collect an independent sample $s_t(s,a)~P(\cdot \mid s,a)$ for every anchor state-action pair $(s,a)\in \mathcal{K}$ and define $Q_{\mathcal{K}}^{(t)}:\mathcal{K}\to ℝ$ to be

$$Q_{\mathcal{K}}^{(t)}(s,a)≔\underset{a^{\prime }\in \mathcal{A}}{\text{max}}{Q}_t\left(s_t,a^{\prime }\right),s_t\equiv s_t(s,a)~P(\cdot \mid s,a).$$

Then given the learning rate η_t ∈ (0, 1], the algorithm adopts the following update rule to update all entries of the Q-function estimate

$$Q_t=\left(1-{\eta }_t\right)Q_{t-1}+{\eta }_t{\mathcal{T}}_{\mathcal{K}}^{(t)}\left(Q_{t-1}\right).$$

Here, ${\mathcal{T}}_{\mathcal{K}}^{(t)}$ is an empirical Bellman operator associated with the linear transition model M and the set $\mathcal{K}$ and is given by

$${\mathcal{T}}_{\mathcal{K}}^{(t)}(Q)(s,a)≔r(s,a)+\gamma \lambda (s,a)Q_{\mathcal{K}}^{(t)},$$

where (2.3) is used in the construction. Clearly, this newly defined operator ${\mathcal{T}}_{\mathcal{K}}^{(t)}$ is an unbiased estimate of the famous Bellman operator $\mathcal{T}$ [Bel52] defined as

$$\forall (s,a)\in \mathcal{S}\times \mathcal{A} : \mathcal{T}(Q)(s,a)≔r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }~P(\cdot \mid s,a)}\left[\underset{a^{\prime }\in \mathcal{A}}{\text{max}}Q\left(s^{\prime },a^{\prime }\right)\right].$$

A critical property is that the Bellman operator $\mathcal{T}$ is contractive with a unique fixed point which is the optimal Q-function Q⋆ [Bel52]. To solve the fixed-point equation $\mathcal{T}\left(Q^{\star }\right)=Q^{\star }$, Q-learning was then introduced by [WD92] based on the idea of stochastic approximation [RM51]. This procedure is precisely described in Algorithm 2.

4.2. Main results

We are now ready to provide our main result for vanilla Q-learning, assuming sampling access to a generative model.

Theorem 2.

Consider any δ ∈ (0, 1) and ε ∈ (0, 1]. Assume that for any 0 ≤ t ≤ T, the learning rates satisfy

$$\frac{1}{1+\frac{c_1(1-\gamma )T}{{\text{log}}^{2}T}}\le {\eta }_t\le \frac{1}{1+\frac{c_2(1-\gamma )t}{{\text{log}}^{2}T}}$$ (4.1)

for some sufficiently small universal constants c₁ ≥ c₂ > 0. Suppose that the total number of iterations T exceeds

$$T\ge \frac{C_3\text{log}(KT/\delta ){\text{log}}^{4}T}{{(1-\gamma )}^4{\epsilon }^2}$$ (4.2)

for some sufficiently large universal constant C₃ > 0. If the initialization obeys $0\le Q_0(s,a)\le \frac{1}{1-\gamma }$ for any $(s,a)\in \mathcal{S}\times \mathcal{A}$, then with probability exceeding 1 − δ, the output Q_T of Algorithm 2 satisfies

$$\underset{(s,a)\in \mathcal{S}\times \mathcal{A}}{\text{max}}\left|Q_T(s,a)-Q^{\star }(s,a)\right|\le \epsilon .$$ (4.3)

In addition, let π_T (resp. V_T) to be the policy (resp. value function) induced by Q_T, then one has

$$\underset{s\in \mathcal{S}}{\text{max}}\left|V^{{\pi }_T}(s)-V^{\star }(s)\right|\le \frac{2\gamma \epsilon }{1-\gamma }.$$ (4.4)

This theorem provides theoretical guarantees on the performance of Algorithm 2. A few implications of this theorem are in order.

• Learning rate. The condition (4.1) accommodates two commonly adopted choice of learning rates: (i) linearly rescaled learning rates η_t = [1 + c₂(1 − γ)t/log²T]⁻¹, and (ii) iteration-invariant learning rates η_t ≡ [1 + c₁(1 − γ)T/log²T]. Interested readers are referred to the discussions in [LCC+21a, Section 3.1] for more details on these two learning rate schemes.

• Tight sample complexity bound. Note that we draw K independent samples in each iteration, therefore the iteration complexity (4.2) can be translated into the sample complexity bound TK in order for Q-learning to achieve ε-accuracy:

  $$\tilde{O}\left(\frac{K}{{(1-\gamma )}^4{\epsilon }^2}\right).$$ (4.5)

  As we will see shortly, this result improves the state-of-the-art sample complexity bound presented in [YW19, Theorem 2]. In addition, the dependency on the effective horizon (1 − γ)⁻⁴ matches the lower bound established in [LCC+21a, Theorem 2] for vanilla Q-learning using either learning rate scheme covered in the previous remark, suggesting that our sample complexity bound (4.5) is sharp.

• Stability vis-à-vis model misspecification. Just like the model-based approach, we can also show that Q-learning is also stable vis-á-vis model misspecification when P approximately admits the linear transition model. We refer interested readers to Theorem 4 in Appendix B for more details.

Comparison with [YW19].

We compare our result with the sample complexity bounds for Q-learning under the feature-based linear transition model in [YW19].

• We first compare our result with [YW19, Theorem 2], which is, to the best of our knowledge, the state-of-the-art theory for this problem. When there is no model misspecification, [YW19, Theorem 2] showed that in order for their Phased Parametric Q-learning² (Algorithm 1 therein) to learn an ε-optimal policy, the sample size needs to be

  $$\tilde{O}\left(\frac{K}{{(1-\gamma )}^7{\epsilon }^2}\right).$$

  Note that (4.5) is the sample complexity required for entrywise ε-accurate estimate of the optimal Q-function, thus a fair comparison requires to deduce the sample complexity for learning an ε-optimal policy from (4.4), which is

  $$\tilde{O}\left(\frac{K}{{(1-\gamma )}^6{\epsilon }^2}\right).$$

  Hence, our sample complexity improves upon previous work by a factor at least on the order of (1 − γ)⁻¹. However it is worth mentioning that [YW19, Theorem 2] is built upon weaker conditions $\sum _{i=1}^K{\lambda }_i(s,a)=1$ and ${\sum }_{i=1}^K\left|{\lambda }_i(s,a)\right|\le L$ for some L ≥ 1, which does not require λ_i(s, a) ≥ 0. Our result holds under Assumption 1, which requires ${\sum }_{i=1}^K{\lambda }_i(s,a)=1$ and λ_i(s, a) ≥ 0. Under the current analysis framework, it is difficult to obtain tight sample complexity bounds without assuming λ_i(s, a) ≥ 0.
• Besides vanilla Q-learning, [YW19] also proposed a new variant of Q-learning called Optimal Phased Parametric Q-Learning (Algorithm 2 therein), which is essentially Q-learning with variance reduction. [YW19, Theorem 3] showed that the sample complexity for this algorithm is

  $$\tilde{O}\left(\frac{K}{{(1-\gamma )}^3{\epsilon }^2}\right),$$

  which matches minimax optimal lower bound (up to a logarithm factor) established in [YW19, Theorem 1]. Careful reader might notice that this sample complexity bound is better than ours for vanilla Q-learning. We emphasize that as elucidated in the second implication under Theorem 2, our result is already tight for vanilla Q-learning. This observation reveals that while the sample complexity for vanilla Q-learning is provably sub-optimal, the variants of Q-learning can have better performance and achieve minimax optimal sample complexity.

We conclude this section by comparing model-based and model-free approaches. Theorem 1 shows that the sample complexity of the model-based approach is minimax optimal, whilst vanilla Q-learning, perhaps the most commonly adopted model-free method, is sub-optimal according to Theorem 2. However this does not mean that model-based method is better than model-free ones since (i) some variants of Q-learning (see [YW19, Algorithm 2] for example) also has minimax optimal sample complexity; and (ii) in many applications it might be unrealistic to estimate the model in advance.

5. A glimpse of our technical approaches

The establishment of Theorems 1 and 2 calls for a series of technical novelties in the proof. In what follows, we briefly highlight our key technical ideas and novelties.

• For the model-based approach, we employ “leave-one-out” analysis to decouple the complicated statistical dependency between the empirical probability transition model $\widehat{P}$ and the corresponding optimal policy. Specifically, [AKY20] proposed to construct a collection of auxiliary MDPs where each one of them leaves out a single state s by setting s to be an absorbing state and keeping everything else untouched. We tailor this high level idea to the needs of linear transition model, then the independence between the newly constructed MDP with absorbing state s and data samples collected at state s will facilitate our analysis, as detailed in Lemma 1. This “leave-one-out” type of analysis has been utilized in studying numerous problems by a long line of work, such as [EK18,MWCC18,Wai19a,CFMW19,CCF+20,CFMY20,CFWY21], just to name a few.

• To obtain tighter sample complexity bound than the previous one $\tilde{O}\left(\frac{K}{{(1-\gamma )}^7{\epsilon }^2}\right)$ in [YW19] for vanilla Q-learning, we invoke Freedman’s inequality [Fre75] for the concentration of an error term with martingale structure as illustrated in Appendix C, while the classical ones used in analyzing Q-learning are Hoeffding’s inequality and Bernstein’s inequality [YW19]. The use of Freedman’s inequality helps us establish a recursive relation on ${\left\{{\Vert Q_t-Q^{\star }\Vert }_{\infty }\right\}}_{t=0}^T$, which consequently leads to the performance guarantee (4.3). It is worth mentioning that [LCC+21a] also studied vanilla Q-learning in the tabular MDP setting and adopted Freedman’s inequality, while we emphasize that it requires a lot of efforts and more delicate analyses in order to study linear transition model and also allow for model misspecification in the current paper, as detailed in the appendices.

6. Additional related literature

To remedy the issue of prohibitively high sample complexity, there exists a substantial body of literature proposing and studying many structural assumptions and complexity notions under different settings. This current paper focuses on linear transition model which is studied in MDP by numerous previous works [YW19,JYWJ20,YW20,ZHG21,MJTS20,HDL+21,WDYS20,TV20,HZG21,WJLJ21]. Among them, [YW19] studied linear transition model and provided tight sample complexity bounds for a new variant of Q-learning with the help of variance reduction. [JYWJ20] focused on linear MDP and designed an algorithm called “Least-Squares Value Iteration with UCB” with both polynomial runtime and polynomial sample complexity without accessing generative model. [WDYS20] extended the study of linear MDP to the framework of reward-free reinforcement learning. [ZHG21] considered a different feature mapping called linear kernel MDP and devised an algorithm with polynomial regret bound without generative model. Other popular structure assumptions include: [WVR17] studied fully deterministic transition dynamics; [JKA+17] introduced Bellman rank and proposed an algorithm which needs sample size polynomially dependent on Bellman rank to obtain a near-optimal policy in contextual decision processes; [DLWZ19] assumed that the value function has low variance compared to the mean for all deterministic policy; [MR07,PLT+08,ABA18,ZLKB20] used linear model to approximate the value function; [LCC+21b] assumed that the optimal Q-function can be linearly-parameterized by the features.

Apart from the linear transition model, another notion adopted in this work is the generative model, whose role in discounted MDP has been studied by extensive literature. The concept of generative model was originally introduced by [KS99], and then widely adopted in numerous works, including [Kak03,AMK13, YW19,Wai19b,AKY20,PW20], to name a few. Specifically, it is assumed that a generative model of the studied MDP is available and can be queried for every state-action pair and output the next state. Among previous works, [AMK13] proved that the minimax lower bound on the sample complexity to obtain an ε-optimal policy was $\tilde{\Omega }\left(\frac{|\mathcal{S}||\mathcal{A}|}{{(1-\gamma )}^3{\epsilon }^2}\right)$. [AMK13] also showed that model-based approach can output an ε-optimal value function with near-optimal sample complexity for ε ∈ (0, 1). Then [AKY20] made significant progress on the challenging problem of establishing minimax optimal sample complexity in estimating an ε-optimal policy with the help of “leave-one-out” analysis.

In addition, after being proposed in [Wat89], Q-learning has become the focus of a rich line of research [WD92, BT96, KS99, EDMB03, AMGK11, JAZBJ18, Wai19a, CZD+19, LWC+20b, XG20]. Among them, [CZD+19,LWC+20b,XG20] studied Q-learning in the presence of Markovian data, i.e. a single sample trajectory. In contrast, under the generative setting of Q-learning where a fresh sample can be drawn from the simulator at each iteration, [Wai19b] analyzed a variant of Q-learning with the help of variance reduction, which was proved to enjoy minimax optimal sample complexity $\tilde{O}\left(\frac{|\mathcal{S}\parallel \mathcal{A}|}{{(1-\gamma )}^3{\epsilon }^2}\right)$. Then more recently, [LCC+21a] improved the lower bound of the vanilla Q-learning algorithm in terms of its scaling with $\frac{1}{1-\gamma }$ and proved a matching upper bound $\tilde{O}\left(\frac{|\mathcal{S}||\mathcal{A}|}{{(1-\gamma )}^4{\epsilon }^2}\right)$.

7. Discussion

This paper studies sample complexity of both model-based and model-free RL under a discounted infinite-horizon MDP with feature-based linear transition model. We establish tight sample complexity bounds for both model-based approaches and Q-learning, which scale linearly with the feature dimension K instead of $|\mathcal{S}|\times |\mathcal{A}|$, thus considerably reduce the required sample size for large-scale MDPs when K is relatively small. Our results are sharp, and the sample complexity bound for the model-based approach matches the minimax lower bound. The current work suggests a couple of directions for future investigation, as discussed in detail below.

• Extension to episodic MDPs. An interesting direction for future research is to study linear transition model in episodic MDP. This focus of this work is infinite-horizon discounted MDPs, and hopefully the analysis here can be extended to study the episodic MDP as well ([DB15,DLB17,AOM17,JA18,WDYK20,HZG21]).

• Continuous state and action space. The state and action spaces in this current paper are still assumed to be finite, since the proof relies heavily on the matrix operations. However, we expect that the results can be extended to accommodate continuous state and action space by employing more complicated analysis.

• Accommodating entire range of ε. Since both value functions and Q-functions can take value in [0, (1 − γ)⁻¹], ideally our theory should cover all choices of ε ∈ (0, (1 − γ)⁻¹]. However we require that ε ∈ (0, (1 − γ)^−1/2] in Theorem 1 and ε ∈ (0, 1] in Theorem 2. While most of the prior works like [AKY20,YW19] also impose these restrictions, a recent work [LWC+20a] proposed a perturbed model-based planning algorithm and proved minimax optimal guarantees for any ε ∈ (0, (1 − γ)⁻¹]. While their work only focused on model-based RL under tabular MDP, an interesting future direction is to improve our theory to accommodate any ε ∈ (0, (1 − γ)⁻¹].

• General function approximation. Another future direction is to extend the study to more general function approximation starting from linear structure covered in this paper. There exists a rich body of work proposing and studying different structures, such as linear value function approximation [MR07,PLT+08,ABA18,ZLKB20], linear MDPs with infinite dimensional features [AHKS20], Eluder dimension [WSY20], Bellman rank [JKA+17] and Witness rank [SJK+19], etc. Therefore, it is hopeful to investigate these settings and improve the sample efficiency.

APP1

A. Notations

In this section we gather the notations that will be used throughout the appendix.

For any vectors $u={\left[u_i\right]}_{i=1}^n\in {ℝ}^n$ and $v={\left[u_i\right]}_{i=1}^n\in {ℝ}^n$, let $u\circ v={\left[u_i{v}_i\right]}_{i=1}^n$ denote the Hadamard product of u and v. We slightly abuse notations to use $\sqrt{\cdot }$ and | · | to define entry-wise operation, i.e. for any vector $v={\left[v_i\right]}_{i=1}^n$ denote $\sqrt{v}≔{\left[\sqrt{v_i}\right]}_{i=1}^n$ and $|v|≔{\left[\left|v_i\right|\right]}_{i=1}^n$. Furthermore, the binary notations ≤ and ≥ are both defined in entry-wise manner, i.e. u ≤ v (resp. u ≥ v) means u_i ≤ v_i (resp. u_i ≥ v_i) for all 1 ≤ i ≤ n. For a collection of vectors $v_1,\cdots ,v_m\in {ℝ}^n$ with $v_i={\left[v_{i,j}\right]}_{j=1}^n\in {ℝ}^n$, we define the max operator to be ${\text{max}}_{1\le i\le m}{v}_i≔{\left[{\text{max}}_{1\le i\le m}{v}_{i,j}\right]}_{j=1}^n$.

For any matrix $M\in {ℝ}^{m\times n}$, ∥M∥₁ is defined as the largest row-wise ℓ₁ norm of M, i.e. $\Vert M\Vert {}_1≔{\text{max}}_i{\sum }_j\left|M_{i,j}\right|$. In addition, we define 1 to be a vector with all the entries being 1, and I be the identity matrix. To express the probability transition function P in matrix form, we define the matrix $P\in {ℝ}^{|\mathcal{S}||\mathcal{A}|\times |\mathcal{S}|}$ to be a matrix whose (s, a)-th row P_s,a corresponds to P(·|s, a). In addition, we define P^π to be the probability transition matrix induced by policy π, i.e. $P_{(s,a),\left(s^{\prime },a^{\prime }\right)}^{\pi }=P_{s,a}\left(s^{\prime }\right){\mathbb{1}}_{\pi \left(s^{\prime }\right)=a^{\prime }}$ for all state-action pairs (s, a) and (s′, a′). We define π_t to be the policy induced by Q_t, i.e. Q_t(s, π_t(s)) = max_a Q_t(s, a) for all $s\in \mathcal{S}$. Furthermore, we denote the reward function r by vector $r\in {ℝ}^{|\mathcal{S}||\mathcal{A}|}$, i.e. the (s, a)-th element of r equals r(s, a). In the same manner, we define $V^{\pi }\in {ℝ}^{|\mathcal{S}|}$, $V^{\star }\in {ℝ}^{|\mathcal{S}|}$, $V_t\in {ℝ}^{|\mathcal{S}|}$, $Q^{\pi }\in {ℝ}^{|\mathcal{S}||\mathcal{A}|}$, $Q^{\star }\in {ℝ}^{|\mathcal{S}\parallel \mathcal{A}|}$ and $Q_t\in {ℝ}^{|\mathcal{S}||\mathcal{A}|}$ to represent V^π, V^⋆, V_t, Q^π, Q^⋆ and Q_t respectively. By using these notations, we can rewrite the Bellman equation as

$$Q^{\pi }=r+\gamma P{V}^{\pi }=r+\gamma P^{\pi }{Q}^{\pi }.$$ (A.1)

Further, for any vector $V\in {ℝ}^{|\mathcal{S}|}$, let ${\text{Var}}_P(V)\in {ℝ}^{|\mathcal{S}||\mathcal{A}|}$ be

$${\text{Var}}_P(V)≔P(V\circ V)-(PV)\circ (PV),$$ (A.2)

and define ${\text{Var}}_{P_{s,a}}(V)\in ℝ$ to be

$${\text{Var}}_{P_{s,a}}(V)≔P_{s,a}(V\circ V)-{\left(P_{s,a}V\right)}^2,$$ (A.3)

where P_s,a is the (s, a)-th row of P.

Next, we reconsider Assumption 1. For any state-action pair (s, a), we define vector $\lambda (s,a)\in {ℝ}^K\left(\text{resp}.\varphi (s,a)\in {ℝ}^K\right)$ with $\lambda (s,a)={\left[{\lambda }_i(s,a)\right]}_{i=1}^K\left(\text{resp}.\varphi (s,a)={\left[{\varphi }_i(s,a)\right]}_{i=1}^K\right)$ and matrix $\Lambda \in {ℝ}^{|\mathcal{S}||\mathcal{A}|\times K}(\text{resp. }\Phi \in {ℝ}^{|\mathcal{S}||\mathcal{A}|\times K})$ whose (s, a)-th row corresponds to λ(s, a)^⊤ (resp. ϕ(s, a)^⊤). Define vector $\psi (s,a)\in {ℝ}^K$ with $\psi (s,a)={\left[{\psi }_i(s,a)\right]}_{i=1}^K$ and matrix $\Psi \in {ℝ}^{K\times |\mathcal{S}|}$ whose (s, a)-th column corresponds to ψ(s, a)^⊤. Further, let $P_{\mathcal{K}}\in {ℝ}^{K\times |\mathcal{S}|}\left(\text{resp. }{\Phi }_{\mathcal{K}}\in {ℝ}^{K\times K}\right)$ to be a submatrix of P (resp. Φ) formed by concatenating the rows $\left\{P_{s,a},(s,a)\in \mathcal{K}\right\}\left(\text{resp}.\left\{{\Phi }_{s,a},(s,a)\in \mathcal{K}\right\}\right)$. By using the previous notations, we can express the relations in Definition 1 and Assumption 1 as $P_{\mathcal{K}}={\Phi }_{\mathcal{K}}\Psi$, P = ΦΨ and $\Phi =\Lambda {\Phi }_{\mathcal{K}}$. Note that Assumption 1 suggests ${\Phi }_{\mathcal{K}}$ is invertible. Taking these equations collectively yields

$$P=\Phi \Psi =\Phi {\Phi }_{\mathcal{K}}^{-1}{P}_{\mathcal{K}}=\Lambda {\Phi }_{\mathcal{K}}{\Phi }_{\mathcal{K}}^{-1}{P}_{\mathcal{K}}=\Lambda P_{\mathcal{K}},$$ (A.4)

which is reminiscent of the anchor word condition in topic modelling [AGM12]. In addition, for each iteration t, we denote the collected samples as ${\left\{s_t(s,a)\right\}}_{(s,a)\in \mathcal{K}}$ and define a matrix ${\widehat{P}}_{\mathcal{K}}^{(t)}\in {\{0,1\}}^{K\times |\mathcal{S}|}$ to be

$${\widehat{P}}_{\mathcal{K}}^{(t)}\left((s,a),s^{\prime }\right)≔\{\begin{array}{ll}1,\hfill & \text{if }{s}^{\prime }=s_t(s,a)\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}$$ (A.5)

for any $(s,a)\in \mathcal{K}$ and $s^{\prime }\in \mathcal{S}$. Further, we define ${\widehat{P}}_t=\Lambda {\widehat{P}}_{\mathcal{K}}^{(t)}$. Then it is obvious to see that ${\widehat{P}}_t$ has nonnegative entries and unit ℓ₁ norm for each row due to Assumption 1, i.e. ${\Vert {\widehat{P}}_t\Vert }_1=1$.

B. Analysis of model-based RL (Proof of Theorem 1)

In this section, we will provide complete proof for Theorem 1. As a matter of fact, our proof strategy here justifies a more general version of Theorem 1 that accounts for model misspecification, as stated below.

Theorem 3.

Suppose that δ > 0 and ε ∈ (0, (1 − γ)^−1/2]. Assume that there exists a probability transition model $\tilde{P}$ obeying Definition 2.1 and Assumption 1 with feature vectors ${\{\varphi (s,a)\}}_{(s,a)\in \mathcal{S}\times \mathcal{A}}\subset {ℝ}^K$ and anchor state-action pairs $\mathcal{K}$ such that

$$\parallel \tilde{P}-P{\parallel }_1\le \xi$$

for some ξ ≥ 0. Let $\widehat{\pi }$ be the policy returned by Algorithm 1. Assume that

$$N\ge \frac{C\text{log}(K/((1-\gamma )\delta ))}{{(1-\gamma )}^3{\epsilon }^2}$$ (B.1)

for some sufficiently large constant C > 0. Then with probability exceeding 1 − δ,

$$Q^{\star }(s,a)-Q^{\widehat{\pi }}(s,a)\le \epsilon +\frac{4{\epsilon }_{\text{opt}}}{1-\gamma }+\frac{22\xi }{{(1-\gamma )}^2},$$ (B.2)

for every state-action pair $(s,a)\in \mathcal{S}\times \mathcal{A}$.

Theorem 3 subsumes Theorem 1 as a special case with ξ = 0. The remainder of this section is devoted to proving Theorem 3.

B.1. Proof of Theorem 3

The err $Q^{\widehat{\pi }}-Q^{\star }$ can be decomposed as

$$\begin{gathered} Q^{\widehat{\pi }}-Q^{\star }=Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}+{\widehat{Q}}^{\widehat{\pi }}-{\widehat{Q}}^{\star }+{\widehat{Q}}^{\star }-Q^{\star } \\ \ge Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}+{\widehat{Q}}^{\widehat{\pi }}-{\widehat{Q}}^{\star }+{\widehat{Q}}^{{\pi }^{\star }}-Q^{\star } \\ \ge -\left({\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }+{\Vert {\widehat{Q}}^{\widehat{\pi }}-{\widehat{Q}}^{\star }\Vert }_{\infty }+{\Vert {\widehat{Q}}^{{\pi }^{\star }}-Q^{\star }\Vert }_{\infty }\right)1. \end{gathered}$$ (B.3)

For policy $\widehat{\pi }$ satisfying the condition in Theorem 1, we have ${\Vert {\widehat{Q}}^{\widehat{\pi }}-{\widehat{Q}}^{\star }\Vert }_{\infty }\le {\epsilon }_{\text{opt}}$. It boils down to control ${\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }$ and ${\Vert {\widehat{Q}}^{{\pi }^{\star }}-Q^{\star }\Vert }_{\infty }$.

To begin with, we can use (A.1) to further decompose ${\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }$ as

$$\begin{gathered} {\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }={\Vert {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}r-{\left(I-\gamma {\widehat{P}}^{\widehat{\pi }}\right)}^{-1}r\Vert }_{\infty } \\ ={\Vert {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}\left[\left(I-\gamma {\widehat{P}}^{\widehat{\pi }}\right)-\left(I-\gamma P^{\widehat{\pi }}\right)\right]{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty } \\ ={\Vert \gamma {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}(P-\widehat{P}){\widehat{V}}^{\widehat{\pi }}\Vert }_{\infty } \\ \le {\Vert \gamma {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}(P-\widehat{P}){\widehat{V}}^{\star }\Vert }_{\infty }+{\Vert \gamma {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}(P-\widehat{P})\left({\widehat{V}}^{\widehat{\pi }}-{\widehat{V}}^{\star }\right)\Vert }_{\infty } \\ \le {\Vert \gamma {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}\left|(P-\widehat{P}){\widehat{V}}^{\star }\right|\Vert }_{\infty }+\frac{2\gamma {\epsilon }_{\text{opt}}}{1-\gamma }. \end{gathered}$$ (B.4)

Here the last inequality is due to

$$\begin{gathered} {\Vert \gamma {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}(P-\widehat{P})\left({\widehat{V}}^{\widehat{\pi }}-{\widehat{V}}^{\star }\right)\Vert }_{\infty } \\ \le \gamma {\Vert {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}\Vert }_1{\Vert (P-\widehat{P})\left({\widehat{V}}^{\widehat{\pi }}-{\widehat{V}}^{\star }\right)\Vert }_{\infty } \\ \le \gamma {\Vert {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}\Vert }_1\left(\Vert P{\Vert }_1+\Vert \widehat{P}{\Vert }_1\right){\Vert {\widehat{V}}^{\widehat{\pi }}-{\widehat{V}}^{\star }\Vert }_{\infty } \\ \le \frac{2\gamma {\epsilon }_{\text{opt}}}{1-\gamma }, \end{gathered}$$

where we use the fact that ${\Vert {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}\Vert }_1\le 1/(1-\gamma )$ and $\Vert P{\Vert }_1=\Vert \widehat{P}{\Vert }_1=1$.

Similarly, for the term ${\Vert {\widehat{Q}}^{{\pi }^{\star }}-Q^{\star }\Vert }_{\infty }$ in (B.3), we have

$${\Vert {\widehat{Q}}^{{\pi }^{\star }}-Q^{\star }\Vert }_{\infty }={\Vert \gamma {\left(I-\gamma P^{{\pi }^{\star }}\right)}^{-1}(P-\widehat{P}){\widehat{V}}^{{\pi }^{\star }}\Vert }_{\infty }\le {\Vert \gamma {\left(I-\gamma P^{{\pi }^{\star }}\right)}^{-1}\left|(P-\widehat{P}){\widehat{V}}^{{\pi }^{\star }}\right|\Vert }_{\infty }.$$ (B.5)

As can be seen from (B.4) and (B.5), it boils down to bound $\left|(P-\widehat{P}){\widehat{V}}^{\star }\right|$ and $\left|(P-\widehat{P}){\widehat{V}}^{{\pi }^{\star }}\right|$. We have the following lemma.

Lemma 1.

With probability exceeding 1 − δ, one has

$$\begin{gathered} \left|{(\mathit{P}-\widehat{\mathit{P}})}_{s,a}{\widehat{V}}^{\star }\right|\le \frac{10\xi }{1-\gamma }+4\sqrt{\frac{2\text{log}(4K/\delta )}{N}}+\frac{4\text{log}(8K/((1-\gamma )\delta ))}{(1-\gamma )N} \\ +\sqrt{\frac{4\text{log}(8K/((1-\gamma )\delta ))}{N}}\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)}, \end{gathered}$$ (B.6)

$$\begin{gathered} \left|{(\mathit{P}-\widehat{\mathit{P}})}_{s,a}{\widehat{V}}^{{\pi }^{\star }}\right|\le \frac{10\xi }{1-\gamma }+4\sqrt{\frac{2\text{log}(4K/\delta )}{N}}+\frac{4\text{log}(8K/((1-\gamma )\delta ))}{(1-\gamma )N} \\ +\sqrt{\frac{4\text{log}(8K/((1-\gamma )\delta ))}{N}}\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{{\pi }^{\star }}\right)}. \end{gathered}$$ (B.7)

Proof.

See Appendix B.2. □

Applying (B.6) to (B.4) reveals that

$$\begin{gathered} {\Vert {\mathit{Q}}^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }\le \sqrt{\frac{4\text{log}(8K/((1-\gamma )\delta ))}{N}}{\Vert \gamma {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)}\Vert }_{\infty } \\ +\frac{\gamma }{1-\gamma }\left[4\sqrt{\frac{2\text{log}(4K/\delta )}{N}}+\frac{4\text{log}(8K/((1-\gamma )\delta ))}{(1-\gamma )N}\right]+\frac{10\gamma \xi }{{(1-\gamma )}^2}+\frac{2\gamma {\epsilon }_{\text{opt}}}{1-\gamma }. \end{gathered}$$ (B.8)

For the first term, one has

$$\begin{gathered} \sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)}\le \sqrt{{\text{Var}}_{P_{s,a}}\left(V^{\widehat{\pi }}\right)}+\sqrt{{\text{Var}}_{P_{s,a}}\left(V^{\widehat{\pi }}-{\widehat{V}}^{\widehat{\pi }}\right)}+\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\widehat{\pi }}-{\widehat{V}}^{\star }\right)} \\ \le \sqrt{{\text{Var}}_{P_{s,a}}\left(V^{\widehat{\pi }}\right)}+{\Vert V^{\widehat{\pi }}-{\widehat{V}}^{\widehat{\pi }}\Vert }_{\infty }+{\epsilon }_{\text{opt}} \\ \le \sqrt{{\text{Var}}_{P_{s,a}}\left(V^{\widehat{\pi }}\right)}+{\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }+{\epsilon }_{\text{opt}}, \end{gathered}$$

where the first inequality comes from the fact that $\sqrt{\text{Var}(X+Y)}\le \sqrt{\text{Var}(X)}+\sqrt{\text{Var}(Y)}$ for any random variables X and Y. It follows that

$$\begin{gathered} {\Vert \gamma {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)}\Vert }_{\infty }\le {\Vert \gamma {\left(I-\gamma P^{\widehat{\pi }}\right)}^{-1}\sqrt{{\text{Var}}_{P_{a,a}}\left(V^{\widehat{\pi }}\right)}\Vert }_{\infty }+\frac{\gamma }{1-\gamma }\left({\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }+{\epsilon }_{\text{opt}}\right) \\ \le \gamma \sqrt{\frac{2}{{(1-\gamma )}^3}}+\frac{\gamma }{1-\gamma }\left({\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }+{\epsilon }_{\text{opt}}\right), \end{gathered}$$ (B.9)

where the second inequality utilizes [AMK13, Lemma 7].

Plugging (B.9) into (B.8) yields

$$\begin{gathered} {\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }\le \sqrt{\frac{4\text{log}(8K/((1-\gamma )\delta ))}{N}}\left[\gamma \sqrt{\frac{2}{{(1-\gamma )}^3}}+\frac{\gamma }{1-\gamma }\left({\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }+{\epsilon }_{\text{opt}}\right)\right] \\ +\frac{\gamma }{1-\gamma }\left[4\sqrt{\frac{2\text{log}(4K/\delta )}{N}}+\frac{4\text{log}(8K/((1-\gamma )\delta ))}{(1-\gamma )N}\right]+\frac{10\gamma \xi }{{(1-\gamma )}^2}+\frac{2\gamma {\epsilon }_{\text{opt}}}{1-\gamma }. \end{gathered}$$

Then we can rearrange terms to obtain

$${\Vert Q^{\widehat{\pi }}-{\widehat{Q}}^{\widehat{\pi }}\Vert }_{\infty }\le 10\gamma \sqrt{\frac{\text{log}(8K/((1-\gamma )\delta ))}{N{(1-\gamma )}^3}}+\frac{11\gamma \xi }{{(1-\gamma )}^2}+\frac{3\gamma {\epsilon }_{\text{opt}}}{1-\gamma }$$ (B.10)

as long as $N\ge \tilde{C}\text{log}(8K/((1-\gamma )\delta ))/{(1-\gamma )}^2$ for some sufficiently large constant $\tilde{C}>0$.

In a similar vein, we can use (B.5) and (B.7) to obtain that

$${\Vert {\widehat{Q}}^{{\pi }^{\star }}-Q^{\star }\Vert }_{\infty }\le 10\gamma \sqrt{\frac{\text{log}(8K/((1-\gamma )\delta ))}{N{(1-\gamma )}^3}}+\frac{11\gamma \xi }{{(1-\gamma )}^2}.$$ (B.11)

Finally, we can substitute (B.10) and (B.11) into (B.3) to achieve

$$Q^{\widehat{\pi }}-Q^{\star }\ge -\left(20\gamma \sqrt{\frac{\text{log}(8K/((1-\gamma )\delta ))}{N{(1-\gamma )}^3}}+\frac{22\gamma \xi }{{(1-\gamma )}^2}+\frac{3\gamma {\epsilon }_{\text{opt}}}{1-\gamma }+{\epsilon }_{\text{opt}}\right)1.$$

This result implies that

$$Q^{\widehat{\pi }}\ge Q^{\star }-\left(\epsilon +\frac{22\xi }{{(1-\gamma )}^2}+\frac{4{\epsilon }_{\text{opt}}}{1-\gamma }\right)1,$$

as long as

$$N\ge \frac{C\text{log}(8K/((1-\gamma )\delta ))}{{(1-\gamma )}^3{\epsilon }^2},$$

for some sufficiently large constant C > 0.

B.2. Proof of Lemma 1

To prove this theorem, we invoke the idea of s-absorbing MDP proposed by [AKY20]. For a state $s\in \mathcal{S}$ and a scalar u, we define a new MDP M_s,u to be identical to M on all the other states except s; on state s, M_s,u is absorbing such that $P_{M_{s,u}}(s\mid s,a)=1$ and $r_{M_{s,u}}(s,a)=(1-\gamma )u$ for all $a\in \mathcal{A}$. More formally, we define $P_{M_{u,s}}$ and $r_{M_{u,s}}$ as

$$\begin{array}{ll}\hfill P_{M_{s,u}}(s\mid s,a)=1, r_{M_{s,u}}(s,a)=(1-\gamma )u,& \text{for all }a\in \mathcal{A},\hfill \\ \hfill P_{M_{s,u}}\left(\cdot \mid s^{\prime },a^{\prime }\right)=P\left(\cdot \mid s^{\prime },a^{\prime }\right), r_{M_{s,u}}(s,a)=r(s,a),& \text{for all }{s}^{\prime }\ne s\text{ and }{a}^{\prime }\in \mathcal{A}.\hfill \end{array}$$

To streamline notations, we will use $V_{s,u}^{\pi }\in {ℝ}^{|\mathcal{S}|}$ and $V_{s,u}^{\star }\in {ℝ}^{|\mathcal{S}|}$ to denote the value function of M_s,u under policy π and the optimal value function of M_s,u respectively. Furthermore, we denote by ${\widehat{M}}_{s,u}$ the MDP whose probability transition kernel is identical to $\widehat{P}$ at all states except that state s is absorbing. Similar as before, we use ${\widehat{V}}_{s,u}^{\star }\in {ℝ}^{|\mathcal{S}|}$ to denote the optimal value function under ${\widehat{M}}_{s,u}$. The construction of this collection of auxiliary MDPs will facilitate our analysis by decoupling the statistical dependency between $\widehat{P}$ and ${\widehat{\pi }}^{\star }$.

To begin with, we can decompose the quantity of interest as

$$\begin{gathered} \left|{(P-\widehat{P})}_{s,a}{\widehat{V}}^{\star }\right|=\left|{(P-\widehat{P})}_{s,a}\left({\widehat{V}}^{\star }-{\widehat{V}}_{s,u}^{\star }+{\widehat{V}}_{s,u}^{\star }\right)\right| \\ \le \left|{(P-\widehat{P})}_{s,a}{\widehat{V}}_{s,u}^{\star }\right|+\left|{(P-\widehat{P})}_{s,a}\left({\widehat{V}}^{\star }-{\widehat{V}}_{s,u}^{\star }\right)\right| \\ \stackrel{(\text{i})}{\le }\left|{(P-\tilde{P})}_{s,a}{\widehat{V}}_{s,u}^{\star }\right|+\left|\lambda (s,a)\left({\tilde{P}}_{\mathcal{K}}-P_{\mathcal{K}}\right){\widehat{V}}_{s,u}^{\star }\right| \\ +\left|\lambda (s,a)\left(P_{\mathcal{K}}-{\widehat{P}}_{\mathcal{K}}\right){\widehat{V}}_{s,u}^{\star }\right|+\left({\Vert P_{s,a}\Vert }_1+{\Vert {\widehat{P}}_{s,a}\Vert }_1\right){\Vert {\widehat{V}}^{\star }-{\widehat{V}}_{s,u}^{\star }\Vert }_{\infty } \end{gathered}$$ (B.12)

$$\begin{gathered} \le {\Vert {(P-\tilde{P})}_{s,a}\Vert }_1{\Vert {\widehat{V}}_{s,u}^{\star }\Vert }_{\infty }+\Vert \lambda (s,a){\Vert }_1\cdot {\Vert \left({\tilde{P}}_{\mathcal{K}}-P_{\mathcal{K}}\right){\widehat{V}}_{s,u}^{\star }\Vert }_{\infty } \\ +\Vert \lambda (s,a){\Vert }_1\cdot {\Vert \left(P_{\mathcal{K}}-{\widehat{P}}_{\mathcal{K}}\right){\widehat{V}}_{s,u}^{\star }\Vert }_{\infty }+2{\Vert {\widehat{V}}^{\star }-{\widehat{V}}_{s,u}^{\star }\Vert }_{\infty } \end{gathered}$$ (B.13)

$$\stackrel{\text{ (ii) }}{\le }\frac{2\xi }{1-\gamma }+\underset{(s,a)\in \mathcal{K}}{\text{max}}\left|{(P-\widehat{P})}_{s,a}{\widehat{V}}_{s,u}^{\star }\right|+2{\Vert {\widehat{V}}^{\star }-{\widehat{V}}_{s,u}^{\star }\Vert }_{\infty },$$ (B.14)

where (i) makes use of ${\tilde{P}}_{s,a}=\lambda (s,a){\tilde{P}}_{\mathcal{K}}$ and ${\widehat{P}}_{s,a}=\lambda (s,a){\widehat{P}}_{\mathcal{K}}$; (ii) depends on $\Vert P-\tilde{P}{\Vert }_1\le \xi$, ∥λ(s, a)∥₁ = 1 and ${\Vert {\widehat{V}}_{s,u}^{\star }\Vert }_{\infty }\le {(1-\gamma )}^{-1}$. For each state s, the value of u will be selected from a set ${\mathcal{U}}_s$. The choice of ${\mathcal{U}}_s$ will be specified later. Then for some fixed u in ${\mathcal{U}}_s$ and fixed state-action pair $(s,a)\in \mathcal{K}$, due to the independence between ${\widehat{P}}_{s,a}$. and ${\widehat{V}}_{s,u}^{\star }$, we can apply Bernstein’s inequality (cf. [Ver18, Theorem 2.8.4]) conditional on ${\widehat{V}}_{s,u}^{\star }$ to reveal that with probability greater than 1 − δ/2,

$$\left|{(P-\widehat{P})}_{s,a}{\widehat{V}}_{s,u}^{\star }\right|\le \sqrt{\frac{2\text{log}(4/\delta )}{N}{\text{Var}}_{P_{s,a}}\left({\widehat{V}}_{s,u}^{\star }\right)}+\frac{2\text{log}(4/\delta )}{3(1-\gamma )N}.$$ (B.15)

Invoking the union bound over all the K state-action pairs of $\mathcal{K}$ and all the possible values of u in ${\mathcal{U}}_s$ demonstrate that with probability greater than 1 − δ/2,

$$\left|{(P-\widehat{P})}_{s,a}{\widehat{V}}_{s,u}^{\star }\right|\le \sqrt{\frac{2\text{log}\left(4K\left|{\mathcal{U}}_s\right|/\delta \right)}{N}{\text{Var}}_{P_{s,a}}\left({\widehat{V}}_{s,u}^{\star }\right)}+\frac{2\text{log}\left(4K\left|{\mathcal{U}}_s\right|/\delta \right)}{3(1-\gamma )N},$$ (B.16)

holds for all state-action pair $(s,a)\in \mathcal{K}$ and all $u\in {\mathcal{U}}_s$. Here, ${\text{Var}}_{P_{s,a}}(\cdot )$ is defined in (A.3). Then we observe that

$$\begin{gathered} \sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}_{s,u}^{\star }\right)}\le \sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }-{\widehat{V}}_{s,u}^{\star }\right)}+\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)} \\ \le {\Vert {\widehat{V}}^{\star }-{\widehat{V}}_{s,u}^{\star }\Vert }_{\infty }+\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)} \\ \le \left|{\widehat{V}}^{\star }(s)-u\right|+\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)}, \end{gathered}$$ (B.17)

where (i) is due to $\sqrt{{\text{Var}}_{P_{s,a}}\left(V_1+V_2\right)}\le \sqrt{{\text{Var}}_{P_{s,a}}\left(V_1\right)}+\sqrt{{\text{Var}}_{P_{s,a}}\left(V_2\right)}$ and (ii) holds since

$${\Vert {\widehat{V}}^{\star }-{\widehat{V}}_{s,u}^{\star }\Vert }_{\infty }={\Vert {\widehat{V}}_{s,{\widehat{V}}^{\star }(s)}^{\star }-{\widehat{V}}_{s,u}^{\star }\Vert }_{\infty }\le \left|{\widehat{V}}^{\star }(s)-u\right|,$$ (B.18)

whose proof can be found in [AKY20, Lemma 8 and 9].

By substituting (B.16), (B.17) and (B.18) into (B.14), we arrive at

$$\begin{gathered} \left|{(P-\widehat{P})}_{s,a}{\widehat{V}}^{\star }\right|\le \frac{2\xi }{1-\gamma }+\left|{\widehat{V}}^{\star }(s)-u\right|\left(2+\sqrt{\frac{2\text{log}\left(4K\left|{\mathcal{U}}_s\right|/\delta \right)}{N}}\right) \\ +\sqrt{\frac{2\text{log}\left(4K\left|{\mathcal{U}}_s\right|/\delta \right)}{N}}\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)}+\frac{2\text{log}\left(4K\left|{\mathcal{U}}_s\right|/\delta \right)}{3(1-\gamma )N}. \end{gathered}$$ (B.19)

Then it boils down to determining ${\mathcal{U}}_s$. The coarse bounds of ${\widehat{Q}}^{{\pi }^{\star }}$ and ${\widehat{Q}}^{\star }$ in the following lemma provide a guidance on the choice of ${\mathcal{U}}_s$.

Lemma 2.

For δ ∈ (0, 1), with probability exceeding 1 − δ/2 one has

$${\Vert Q^{\star }-{\widehat{Q}}^{{\pi }^{\star }}\Vert }_{\infty }\le \frac{\gamma }{1-\gamma }\sqrt{\frac{\text{log}(4K/\delta )}{2N{(1-\gamma )}^2}}+\frac{2\gamma \xi }{{(1-\gamma )}^2},$$ (B.20)

$${\Vert Q^{\star }-{\widehat{Q}}^{\star }\Vert }_{\infty }\le \frac{\gamma }{1-\gamma }\sqrt{\frac{\text{log}(4K/\delta )}{2N{(1-\gamma )}^2}}+\frac{2\gamma \xi }{{(1-\gamma )}^2}.$$ (B.21)

Proof.

See Appendix B.3. □

This inspires us to choose U_s to be the set consisting of equidistant points in [V^⋆ (s) − R(δ), V^⋆ (s)+R(δ)] with |U_s| = ⌈1/(1 – γ)⌉² and

$$R(\delta )≔\frac{\gamma }{1-\gamma }\sqrt{\frac{\text{log}(4K/\delta )}{2N{(1-\gamma )}^2}}+\frac{2\gamma \xi }{{(1-\gamma )}^2}.$$

Since ${\Vert V^{\star }-{\widehat{V}}^{\star }\Vert }_{\infty }\le {\Vert Q^{\star }-{\widehat{Q}}^{\star }\Vert }_{\infty }$, Lemma 2 implies that ${\widehat{V}}^{\star }(s)\in \left[V^{\star }(s)-R(\delta ),V^{\star }(s)+R(\delta )\right]$ with probability over 1 − δ/2. Hence, we have

$$\underset{u\in {\mathcal{U}}_s}{\text{min}}\left|{\widehat{V}}^{\star }(s)-u\right|\le \frac{2R(\delta )}{\left|U_s\right|+1}\le 2\gamma \sqrt{\frac{2\text{log}(4K/\delta )}{N}}+4\gamma \xi .$$ (B.22)

Consequently, with probability exceeding 1 − δ, one has

$$\begin{gathered} \left|{(P-\widehat{P})}_{s,a}{\widehat{V}}^{\star }\right|\stackrel{\text{ (i) }}{\le }\frac{2\xi }{1-\gamma }+\underset{u\in {\mathcal{U}}_s}{\text{min}}\left|{\widehat{V}}^{\star }(s)-u\right|\left(2+\sqrt{\frac{2\text{log}\left(4K\left|{\mathcal{U}}_s\right|/\delta \right)}{N}}\right) \\ +\sqrt{\frac{2\text{log}\left(4K\left|{\mathcal{U}}_s\right|/\delta \right)}{N}}\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)}+\frac{2\text{log}\left(4K\left|{\mathcal{U}}_s\right|/\delta \right)}{3(1-\gamma )N} \\ \stackrel{\text{ (ii) }}{\le }\frac{2\xi }{1-\gamma }+\left(2\gamma \sqrt{\frac{2\text{log}(4K/\delta )}{N}}+4\gamma \xi \right)\left(2+\sqrt{\frac{4\text{log}(8K/((1-\gamma )\delta ))}{N}}\right) \\ +\sqrt{\frac{4\text{log}(8K/((1-\gamma )\delta ))}{N}}\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)}+\frac{2\text{log}(8K/((1-\gamma )\delta ))}{3(1-\gamma )N} \\ \le \frac{10\xi }{1-\gamma }+4\sqrt{\frac{2\text{log}(4K/\delta )}{N}}+\frac{4\text{log}(8K/((1-\gamma )\delta ))}{(1-\gamma )N} \\ +\sqrt{\frac{4\text{log}(8K/((1-\gamma )\delta ))}{N}}\sqrt{{\text{Var}}_{P_{s,a}}\left({\widehat{V}}^{\star }\right)}, \end{gathered}$$

where (i) follows from (B.19) and (ii) utilizes (B.22). This finishes the proof for the first inequality. The second inequality can be proved in a similar way and is omitted here for brevity.

B.3. Proof of Lemma 2

To begin with, one has

$$\begin{gathered} {\Vert (\widehat{P}-P)V^{\star }\Vert }_{\infty }\le {\Vert \Lambda \left({\widehat{P}}_{\mathcal{K}}-P_{\mathcal{K}}\right)V^{\star }\Vert }_{\infty }+{\Vert \Lambda \left(P_{\mathcal{K}}-{\tilde{P}}_{\mathcal{K}}\right)V^{\star }\Vert }_{\infty }+{\Vert (\tilde{P}-P)V^{\star }\Vert }_{\infty } \\ \le \Vert \Lambda {\Vert }_1{\Vert \left({\widehat{P}}_{\mathcal{K}}-P_{\mathcal{K}}\right)V^{\star }\Vert }_{\infty }+\Vert \Lambda {\Vert }_1{\Vert \left(P_{\mathcal{K}}-{\tilde{P}}_{\mathcal{K}}\right)V^{\star }\Vert }_{\infty }+\Vert \tilde{P}-P{\Vert }_1{\Vert V^{\star }\Vert }_{\infty } \\ \le {\Vert \left({\widehat{P}}_{\mathcal{K}}-P_{\mathcal{K}}\right)V^{\star }\Vert }_{\infty }+\frac{2\xi }{1-\gamma }, \end{gathered}$$ (B.23)

where the first line uses $\widehat{P}=\Lambda {\widehat{P}}_{\mathcal{K}}$ and $\tilde{P}=\Lambda {\tilde{P}}_{\mathcal{K}}$; the last inequality comes from the facts that $\Vert \tilde{P}-P{\Vert }_1\le \xi ,$, ∥Λ∥₁ = 1 and ∥V^⋆∥_∞ ≤ (1 − γ)⁻¹. Then we turn to bound ${\Vert \left({\widehat{P}}_{\mathcal{K}}-P_{\mathcal{K}}\right)V^{\star }\Vert }_{\infty }$. In view of (3.1), Hoeffding’s inequality (cf. [Ver18, Theorem 2.2.6]) implies that for $(s,a)\in \mathcal{K}$,

$$\mathbb{P}\left(\left|{(\widehat{P}-P)}_{s,a}{V}^{\star }\right|\ge t\right)\le 2\text{exp}\left(-\frac{2t^2}{{\Vert V^{\star }\Vert }_{\infty }^2/N}\right).$$

Hence by the standard union bound argument we have

$${\Vert \left({\widehat{P}}_{\mathcal{K}}-P_{\mathcal{K}}\right)V^{\star }\Vert }_{\infty }\le \sqrt{\frac{{\Vert V^{\star }\Vert }_{\infty }^2\text{log}(4K/\delta )}{2N}}\le \sqrt{\frac{\text{log}(4K/\delta )}{2N{(1-\gamma )}^2}},$$ (B.24)

with probability over 1 – δ/2.

1. Now we are ready to bound $Q^{{\pi }^{\star }}-{\widehat{Q}}^{{\pi }^{\star }}$. One has

   $$\begin{gathered} Q^{{\pi }^{\star }}-{\widehat{Q}}^{{\pi }^{\star }}={\left(I-\gamma P^{{\pi }^{\star }}\right)}^{-1}r-{\left(I-\gamma {\widehat{P}}^{{\pi }^{\star }}\right)}^{-1}r \\ ={\left(I-\gamma {\widehat{P}}^{{\pi }^{\star }}\right)}^{-1}\left(\left(I-\gamma {\widehat{P}}^{{\pi }^{\star }}\right)-\left(I-\gamma P^{{\pi }^{\star }}\right)\right)Q^{{\pi }^{\star }} \\ =\gamma {\left(I-\gamma {\widehat{P}}^{{\pi }^{\star }}\right)}^{-1}\left(P^{{\pi }^{\star }}-{\widehat{P}}^{{\pi }^{\star }}\right)Q^{{\pi }^{\star }} \\ =\gamma {\left(I-\gamma {\widehat{P}}^{{\pi }^{\star }}\right)}^{-1}(P-\widehat{P})V^{{\pi }^{\star }}, \end{gathered}$$
   where the first equality makes use of (A.1). Then we take (B.23) and (B.24) collectively to achieve

   $$\begin{gathered} {\Vert \gamma {\left(I-\gamma {\widehat{P}}^{{\pi }^{\star }}\right)}^{-1}(P-\widehat{P})V^{\star }\Vert }_{\infty }\le \gamma \sum _{i=0}^{\infty }{\Vert {\gamma }^i{\left({\widehat{P}}^{{\pi }^{\star }}\right)}^i(P-\widehat{P})V^{\star }\Vert }_{\infty } \\ \le \gamma \sum _{i=0}^{\infty }{\gamma }^i{\Vert {\left({\widehat{P}}^{{\pi }^{\star }}\right)}^i\Vert }_1{\Vert (P-\widehat{P})V^{\star }\Vert }_{\infty } \\ \le \frac{\gamma }{1-\gamma }\sqrt{\frac{\text{log}(4K/\delta )}{2N{(1-\gamma )}^2}}+\frac{2\gamma \xi }{{(1-\gamma )}^2}, \end{gathered}$$

   where the last line comes from the fact that for all $i\ge 1,{\left({\widehat{P}}^{{\pi }^{\star }}\right)}^i$ is a probability transition matrix so that ${\Vert {\left({\widehat{P}}^{{\pi }^{\star }}\right)}^i\Vert }_1=1$. This justifies the first inequality (B.20).

2. In terms of the second one, [AKY20, Section A.4] implies that

   $${\Vert Q^{\star }-{\widehat{Q}}^{\star }\Vert }_{\infty }\le \frac{\gamma }{1-\gamma }{\Vert (P-\widehat{P})V^{\star }\Vert }_{\infty }.$$
   Substitution of (B.23) and (B.24) into the above inequality yields

   $${\Vert Q^{\star }-{\widehat{Q}}^{\star }\Vert }_{\infty }\le \frac{\gamma }{1-\gamma }\sqrt{\frac{\text{log}(4K/\delta )}{2N{(1-\gamma )}^2}}+\frac{2\gamma \xi }{{(1-\gamma )}^2}.$$

C. Analysis of Q-learning (Proof of Theorem 2)

In this section, we will provide complete proof for Theorem 2. We actually prove a more general version of Theorem 2 that takes model misspecification into consideration, as stated below.

Theorem 4.

Consider any δ ∈ (0, 1) and ε ∈ (0, 1]. Suppose that there exists a probability transition model $\tilde{P}$ obeying Definition 2.1 and Assumption 1 with feature vectors ${\{\varphi (s,a)\}}_{(s,a)\in \mathcal{S}\times \mathcal{A}}\subset {ℝ}^K$ and anchor state-action pairs $\mathcal{K}$ such that

$$\Vert \tilde{P}-P{\Vert }_1\le \xi$$

for some ξ ≥ 0. Assume that the initialization obeys $0\le Q_0(s,a)\le \frac{1}{1-\gamma }$ for any $(s,a)\in \mathcal{S}\times \mathcal{A}$ and for any 0 ≤ t ≤ T, the learning rates satisfy

$$\frac{1}{1+\frac{c_1(1-\gamma )T}{{\text{log}}^{2}T}}\le {\eta }_t\le \frac{1}{1+\frac{c_2(1-\gamma )t}{{\text{log}}^{2}T}},$$ (C.1)

for some sufficiently small universal constants c₁ ≥ c₂ > 0. Suppose that the total number of iterations T exceeds

$$T\ge \frac{C_3\text{log}(KT/\delta ){\text{log}}^{4}T}{{(1-\gamma )}^4{\epsilon }^2},$$ (C.2)

for some sufficiently large universal constant C₃ > 0. If there exists a linear probability transition model $\tilde{P}$ satisfying Assumption 1 with feature vectors ${\{\varphi (s,a)\}}_{(s,a)\in \mathcal{S}\times \mathcal{A}}$ such that $\Vert \tilde{P}-P{\Vert }_1\le \xi$, then with probability exceeding 1 − δ, the output Q_T of Algorithm 2 satisfies

$$\underset{(s,a)\in \mathcal{S}\times \mathcal{A}}{\text{max}}\left|Q_T(s,a)-Q^{\star }(s,a)\right|\le \epsilon +\frac{6\gamma \xi }{{(1-\gamma )}^2},$$ (C.3)

for some constant C₄ > 0. In addition, let π_T (resp. V_T) to be the policy (resp. value function) induced by Q_T, then one has

$$\underset{s\in \mathcal{S}}{\text{max}}\left|V^{{\pi }_T}(s)-V^{\star }(s)\right|\le \frac{2\gamma }{1-\gamma }\left(\epsilon +\frac{6\gamma \xi }{{(1-\gamma )}^2}\right).$$ (C.4)

Theorem 4 subsumes Theorem 2 as a special case with ξ = 0. The remainder of this section is devoted to proving Theorem 4.

C.1. Proof of Theorem 4

First we show that (C.4) can be easily obtained from (C.3). Since [SY94] gives rise to

$${\Vert V^{{\pi }_T}-V^{\star }\Vert }_{\infty }\le \frac{2\gamma {\Vert V_T-V^{\star }\Vert }_{\infty }}{1-\gamma },$$

we have

$${\Vert V^{{\pi }_T}-V^{\star }\Vert }_{\infty }\le \frac{2\gamma {\Vert Q_T-Q^{\star }\Vert }_{\infty }}{1-\gamma },$$

due to ∥V_T − V^⋆∥_∞ ≤ ∥Q_T − Q^⋆∥_∞. Then (C.4) follows directly from (C.3).

Therefore, we are left to justify (C.3). To start with, we consider the update rule

$$Q_t=\left(1-{\eta }_t\right)Q_{t-1}+{\eta }_t\left(r+\gamma {\widehat{P}}_t{V}_{t-1}\right).$$

By defining the error term Δ_t ≔ Q_t − Q^⋆, we can decompose Δ_t into

$$\begin{gathered} {\Delta }_t=\left(1-{\eta }_t\right)Q_{t-1}+{\eta }_t\left(r+\gamma {\widehat{P}}_t{V}_{t-1}\right)-Q^{\star } \\ =\left(1-{\eta }_t\right)\left(Q_{t-1}-Q^{\star }\right)+{\eta }_t\left(r+\gamma {\widehat{P}}_t{V}_{t-1}-Q^{\star }\right) \\ =\left(1-{\eta }_t\right)\left(Q_{t-1}-Q^{\star }\right)+\gamma {\eta }_t\left({\widehat{P}}_t{V}_{t-1}-P{V}^{\star }\right) \\ =\left(1-{\eta }_t\right){\Delta }_{t-1}+\gamma {\eta }_t\Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{t-1}+\gamma {\eta }_t\Lambda P_{\mathcal{K}}\left(V_{t-1}-V^{\star }\right)+\gamma {\eta }_t\left(\Lambda P_{\mathcal{K}}-P\right)V^{\star }. \end{gathered}$$ (C.5)

Here in the penultimate equality, we make use of Q^⋆ = r + γPV^⋆; and the last equality comes from ${\widehat{P}}_t=\Lambda {\widehat{P}}_{\mathcal{K}}^{(t)}$ which is defined in (A.5). It is straightforward to check that $\Lambda P_{\mathcal{K}}$ is also a probability transition matrix. We denote by $\overline{P}=\Lambda P_{\mathcal{K}}$ hereafter. The third term in the decomposition above can be upper and lower bounded by

$$\overline{P}\left(V_{t-1}-V^{\star }\right)={\overline{P}}^{{\pi }_{t-1}}{Q}_{t-1}-{\overline{P}}^{{\pi }^{\star }}{Q}^{\star }\le {\overline{P}}^{{\pi }_{t-1}}{Q}_{t-1}-{\overline{P}}^{{\pi }_{t-1}}{Q}^{\star }={\overline{P}}^{{\pi }_{t-1}}{\Delta }_{t-1},$$

and

$$\overline{P}\left(V_{t-1}-V^{\star }\right)={\overline{P}}^{{\pi }_{t-1}}{Q}_{t-1}-{\overline{P}}^{{\pi }^{\star }}{Q}^{\star }\ge {\overline{P}}^{{\pi }^{\star }}{Q}_{t-1}-{\overline{P}}^{{\pi }^{\star }}{Q}^{\star }={\overline{P}}^{{\pi }^{\star }}{\Delta }_{t-1}.$$

Plugging these bounds into (C.5) yields

$$\begin{gathered} {\Delta }_t\le \left(1-{\eta }_t\right){\Delta }_{t-1}+\gamma {\eta }_t\Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{t-1}+\gamma {\eta }_t{\overline{P}}^{{\pi }_{t-1}}{\Delta }_{t-1}+\gamma {\eta }_t\left(\Lambda P_{\mathcal{K}}-P\right)V^{\star }, \\ {\Delta }_t\ge \left(1-{\eta }_t\right){\Delta }_{t-1}+\gamma {\eta }_t\Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{t-1}+\gamma {\eta }_t{\overline{P}}^{{\pi }^{\star }}{\Delta }_{t-1}+\gamma {\eta }_t\left(\Lambda P_{\mathcal{K}}-P\right)V^{\star }. \end{gathered}$$

Repeatedly invoking these two recursive relations leads to

$${\Delta }_t\le {\eta }_0^{(t)}{\Delta }_0+\sum _{i=1}^t{\eta }_i^{(t)}\gamma \left({\overline{P}}^{{\pi }_{t-1}}{\Delta }_{t-1}+\Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{t-1}+\left(\Lambda P_{\mathcal{K}}-P\right)V^{\star }\right),$$ (C.6)

$${\Delta }_t\ge {\eta }_0^{(t)}{\Delta }_0+\sum _{i=1}^t{\eta }_i^{(t)}\gamma \left({\overline{P}}^{{\pi }^{\star }}{\Delta }_{t-1}+\Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{t-1}+\left(\Lambda P_{\mathcal{K}}-P\right)V^{\star }\right),$$ (C.7)

where

$${\eta }_i^{(t)}≔\{\begin{array}{ll}{\prod }_{j=1}^t\left(1-{\eta }_j\right),\hfill & \text{ if }i=0,\hfill \\ {\eta }_i{\prod }_{j=i+1}^t\left(1-{\eta }_j\right),\hfill & \text{ if }0<i<t,\hfill \\ {\eta }_t,\hfill & \text{ if }i=t.\hfill \end{array}$$

Here we adopt the same notations as [LCC+21a].

To begin with, we consider the upper bound (C.6). It can be further decomposed as

$$\begin{gathered} {\Delta }_t\le \underset{≕{\theta }_t}{\underbrace{{\eta }_0^{(t)}{\Delta }_0+\sum _{i=1}^{(1-\alpha )t}{\eta }_i^{(t)}\gamma \left({\overline{P}}^{{\pi }_{t-1}}{\Delta }_{t-1}+\Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{t-1}\right)}}+\underset{≕{\nu }_t}{\underbrace{\sum _{i=(1-\alpha )t+1}^t{\eta }_i^{(t)}\gamma \Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{i-1}}} \\ +\underset{≕{\omega }_t}{\underbrace{\sum _{i=1}^t{\eta }_i^{(t)}\gamma \left(\Lambda P_{\mathcal{K}}-P\right)V^{\star }}}+\sum _{i=(1-\alpha )t+1}^t{\eta }_i^{(t)}\gamma {\overline{P}}^{{\pi }_{t-1}}{\Delta }_{i-1}, \end{gathered}$$ (C.8)

where we define α ≔ C₄(1 − γ)/log T for some constant C₄ > 0. Next, we turn to bound θ_t and ν_t respectively for any t satisfying $\frac{T}{c_2\text{log}\frac{1}{1-\gamma }}\le t\le T$ with stepsize choice (4.1).

Bounding ω_t.

It is straightforward to bound

$$\begin{gathered} {\Vert {\omega }_t\Vert }_{\infty }\stackrel{(\text{i})}{=}{\Vert \gamma \left(\Lambda P_{\mathcal{K}}-P\right)V^{\star }\Vert }_{\infty } \\ \stackrel{\text{ (ii) }}{\le }\gamma \left(\Vert \Lambda {\Vert }_1{\Vert \left(P_{\mathcal{K}}-{\tilde{P}}_{\mathcal{K}}\right)V^{\star }\Vert }_{\infty }+{\Vert (\tilde{P}-P)V^{\star }\Vert }_{\infty }\right) \\ \stackrel{\text{ (iii) }}{\le }\frac{2\gamma \xi }{1-\gamma }, \end{gathered}$$

where the first equality comes from the fact that ${\sum }_{i=1}^t{\eta }_i^{(t)}=1$ [LCC+21a, Equation (40)]; the second inequality utilizes $\tilde{P}=\Lambda {\tilde{P}}_{\mathcal{K}}$; the last line uses the facts that ∥Λ∥₁ = 1, ∥V^⋆∥_∞ ≤ (1 − γ)⁻¹ and ${\Vert {\tilde{P}}_{\mathcal{K}}-P_{\mathcal{K}}\Vert }_1\le \Vert \tilde{P}-P{\Vert }_1\le \xi$.

Bounding θ_t.

By similar derivation as Step 1 in [LCC+21a, Appendix A.2], we have

$$\begin{gathered} {\Vert {\theta }_t\Vert }_{\infty }\le {\eta }_0^{(t)}{\Vert {\Delta }_0\Vert }_{\infty }+t\underset{1\le i\le (1-\alpha )t}{\text{max}}{\eta }_i^{(t)}\underset{1\le i\le (1-\alpha )t}{\text{max}}\left({\Vert {\overline{P}}^{{\pi }_{t-1}}{\Delta }_{i-1}\Vert }_{\infty }+{\Vert \Lambda {\widehat{P}}_{\mathcal{K}}^{(t)}{V}_{i-1}\Vert }_{\infty }+{\Vert \Lambda P_{\mathcal{K}}{V}_{i-1}\Vert }_{\infty }\right) \\ \stackrel{\text{ (i) }}{\le }{\eta }_0^{(t)}{\Vert {\Delta }_0\Vert }_{\infty }+t\underset{1\le i\le (1-\alpha )t}{\text{max}}{\eta }_i^{(t)}\underset{1\le i\le (1-\alpha )t}{\text{max}}\left({\Vert {\Delta }_{i-1}\Vert }_{\infty }+2{\Vert V_{i-1}\Vert }_{\infty }\right) \\ \stackrel{\left(\text{ii}\right)}{\le }\frac{1}{2T^2}\cdot \frac{1}{1-\gamma }+\frac{1}{2T^2}\cdot t\cdot \frac{3}{1-\gamma } \\ \le \frac{2}{(1-\gamma )T}, \end{gathered}$$ (C.9)

where (i) is due to the fact that ${\Vert {\overline{P}}^{{\pi }_{t-1}}\Vert }_1={\Vert \Lambda {\widehat{P}}_{\mathcal{K}}^{(t)}\Vert }_1={\Vert \Lambda P_{\mathcal{K}}\Vert }_1=1$ and (ii) comes from [LCC+21a, Equation (39a)].

Bounding ν_t.

To control the second term, we apply the following Freedman’s inequality.

Lemma 3 (Freedman’s Inequality).

Consider a real-valued martingale {Y_k : k = 0, 1, 2, ⋯} with difference sequence {X_k : k = 1, 2, 3, ⋯}. Assume that the difference sequence is uniformly bounded:

$$\left|X_k\right|\le R and \mathbb{E}\left[X_k\mid {\left\{X_j\right\}}_{j=1}^{k-1}\right]=0 forallk\ge 1.$$

Let

$$S_n≔\sum _{k=1}^n{X}_i, T_n≔\sum _{k=1}^n\text{Var}\left\{X_k\mid {\left\{X_j\right\}}_{j=1}^{k-1}\right\}.$$

Then for any given σ² ≥ 0, one has

$$\mathbb{P}\left(\left|S_n\right|\ge \tau and{T}_n\le {\sigma }^2\right)\le 2\text{exp}\left(-\frac{{\tau }^2/2}{{\sigma }^2+R\tau /3}\right).$$

In addition, suppose that W_n ≤ σ² holds deterministically. For any positive integer K ≥ 1, with probability at least 1 − δ one has

$$\left|S_n\right|\le \sqrt{8\text{max}\left\{T_n,\frac{{\sigma }^2}{2^K}\right\}\text{log}\frac{2K}{\delta }}+\frac{4}{3}R\text{log}\frac{2K}{\delta }.$$

Proof.

See [LCC+21a, Theorem 4]. □

To apply this inequality, we can express ν_t as

$${\nu }_t≔\sum _{i=(1-\alpha )t+1}^t{x}_i,$$

with

$$x_i≔{\eta }_i^{(t)}\gamma \Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{i-1}, and \mathbb{E}\left[x_i\mid V_{i-1},\cdots ,V_0\right]=0.$$ (C.10)

1. In order to calculate bound R in Lemma 3, one has

   $$\begin{gathered} B≔\underset{(1-\alpha )t<t\le t}{\text{max}}{\Vert x_i\Vert }_{\infty }\le \underset{(1-\alpha )t<t\le t}{\text{max}}{\Vert {\eta }_i^{(t)}\Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{i-1}\Vert }_{\infty } \\ \le \underset{(1-\alpha )t<t\le t}{\text{max}}{\eta }_i^{(t)}\left({\Vert \Lambda {\widehat{P}}_{\mathcal{K}}^{(t)}\Vert }_1+{\Vert \Lambda P_{\mathcal{K}}\Vert }_1\right){\Vert V_{i-1}\Vert }_{\infty } \\ \le \underset{(1-\alpha )t<t\le t}{\text{max}}{\eta }_i^{(t)}\cdot \frac{2}{1-\gamma }\le \frac{4{\text{log}}^{4}T}{{(1-\gamma )}^{2}T}, \end{gathered}$$

   where the last inequality comes from [LCC+21a, Eqn (39b)] and the fact that ${\Vert V_{i-1}\Vert }_{\infty }\le \frac{1}{1-\gamma }$.

2. Then regarding the variance term, we claim for the moment that

   $$\begin{gathered} W_t≔\sum _{i=(1-\alpha )t+1}^t\text{diag}\left(\text{Var}\left(x_i\mid V_{i-1},\cdots ,V_0\right)\right) \\ \le {\gamma }^2\sum _{i=(1-\alpha )t+1}^t{\left({\eta }_i^{(t)}\right)}^2{\text{Var}}_{\overline{P}}\left(V_{i-1}\right). \end{gathered}$$ (C.11)
   Then we have

   $$\begin{gathered} W_t\le \underset{(1-\alpha )t\le i\le t}{\text{max}}{\eta }_i^{(t)}\left(\sum _{i=(1-\alpha )t+1}^t{\eta }_i^{(t)}\right)\underset{(1-\alpha )t\le i<t}{\text{max}}{\text{Var}}_{\overline{P}}\left(V_i\right) \\ \le \frac{2{\text{log}}^{4}T}{(1-\gamma )T}\underset{(1-\alpha )t\le i<t}{\text{max}}{\text{Var}}_{\overline{P}}\left(V_i\right), \end{gathered}$$ (C.12)
   where the second line comes from [LCC+21a, Eqns (39b), (40)]. A trivial upper bound for W_t is

   $$\left|W_t\right|\le \frac{2{\text{log}}^{4}T}{(1-\gamma )T}\cdot \frac{1}{{(1-\gamma )}^2}1=\frac{2{\text{log}}^{4}T}{{(1-\gamma )}^{3}T}1,$$

   which uses the fact that ${\text{Var}}_P\left(V_i\right)\le {\Vert V_i\Vert }_{\infty }^2\le 1/{(1-\gamma )}^2$.

Then, we invoke Lemma 3 with $K=\lceil 2{\text{log}}_2\frac{1}{1-\gamma }\rceil$ and apply the union bound argument over $\mathcal{K}$ to arrive at

$$\begin{gathered} \left|{\nu }_t\right|\le \sqrt{8\left(W_t+\frac{{\sigma }^2}{2^K}1\right)\text{log}\frac{8KT\text{log}\frac{1}{1-\gamma }}{\delta }}+\frac{4}{3}B\text{log}\frac{8KT\text{log}\frac{1}{1-\gamma }}{\delta }1 \\ \le \sqrt{8\left(W_t+\frac{2{\text{log}}^{4}T}{(1-\gamma )T}1\right)\text{log}\frac{8KT}{\delta }}+\frac{4}{3}B\text{log}\frac{8KT\text{log}\frac{1}{1-\gamma }}{\delta }1 \\ \le \sqrt{\frac{32{\text{log}}^{4}T}{(1-\gamma )T}\text{log}\frac{8KT}{\delta }\left(\underset{(1-\alpha )t\le i<t}{\text{max}}{\text{Var}}_{\Lambda P_{\mathcal{K}}}\left(V_i\right)+1\right)}+\frac{12{\text{log}}^{4}T}{{(1-\gamma )}^{2}T}\text{log}\frac{8KT}{\delta }1. \end{gathered}$$ (C.13)

Hence if we define

$${\phi }_t≔64\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{(1-\gamma )T}\left(\underset{\frac{t}{2}\le i\le t}{\text{max}}{\text{Var}}_{\overline{P}}\left(V_i\right)+1\right),$$

then (C.9) and (C.13) implies that

$$\left|{\theta }_t\right|+\left|{\nu }_t\right|+\left|{\omega }_t\right|\le \sqrt{{\phi }_t}+\frac{2\gamma \xi }{1-\gamma }1,$$ (C.14)

with probability over 1 − δ for all 2t/3 ≤ k ≤ t, as long as $T\gg {\text{log}}^{4}T\text{log}\frac{KT}{\delta }/{(1-\gamma )}^3$. Therefore, plugging (C.14) into (C.8), we arrive at the recursive relationship

$${\Delta }_t\le \sqrt{{\phi }_t}+\frac{2\gamma \xi }{1-\gamma }1+\sum _{i=(1-\alpha )k+1}^k{\eta }_i^{(k)}\gamma {\overline{P}}^{{\pi }_{i-1}}{\Delta }_{i-1}=\sqrt{{\phi }_t}+\frac{2\gamma \xi }{1-\gamma }1+\sum _{i=(1-\alpha )k}^{k-1}{\eta }_i^{(k)}\gamma {\overline{P}}^{{\pi }_{i-1}}{\Delta }_i.$$

This recursion is expressed in a similar way as [LCC+21a, Eqn. (46)] so we can invoke similar derivation in [LCC+21a, Appendix A.2] to obtain that

$${\Delta }_t\le 30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}\left(1+\underset{\frac{t}{2}\le i<t}{\text{max}}{\Vert {\Delta }_i\Vert }_{\infty }\right)}1+\frac{2\gamma \xi }{{(1-\gamma )}^2}1.$$ (C.15)

Then we turn to (C.7). Applying a similar argument, we can deduce that

$${\Delta }_t\ge -30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}\left(1+\underset{\frac{t}{2}\le i<t}{\text{max}}{\Vert {\Delta }_i\Vert }_{\infty }\right)}1-\frac{2\gamma \xi }{{(1-\gamma )}^2}1.$$ (C.16)

For any t satisfying $\frac{T}{c_2\text{log}\frac{1}{1-\gamma }}\le t\le T$, taking (C.15) and (C.16) collectively gives rise to

$${\Vert {\Delta }_t\Vert }_{\infty }\le 30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}\left(1+\underset{\frac{t}{2}\le i<t}{\text{max}}{\Vert {\Delta }_i\Vert }_{\infty }\right)}+\frac{2\gamma \xi }{{(1-\gamma )}^2}.$$ (C.17)

Let

$$u_k≔\text{max}\left\{{\Vert {\Delta }_t\Vert }_{\infty }:2^k\frac{T}{c_2\text{log}\frac{1}{1-\gamma }}\le t\le T\right\}.$$

By taking supremum over $t\in \left\{\lceil 2^{k}T/\left(c_2\text{log}\frac{1}{1-\gamma }\right)\rceil ,\dots ,T\right\}$ on both sides of (C.17), we have

$$u_k\le 30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}\left(1+u_{k-1}\right)}+\frac{2\gamma \xi }{{(1-\gamma )}^2} \forall 1\le k\le \text{log}\left(c_2\text{log}\frac{1}{1-\gamma }\right).$$ (C.18)

It is straightforward to bound $u_0\le \frac{1}{1-\gamma }$. For k ≥ 1, it is straightforward to obtain from (C.18) that

$$u_k\le 3\text{max}\left\{30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}},30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}{u}_{k-1}},\frac{2\gamma \xi }{{(1-\gamma )}^2}\right\},$$ (C.19)

for $1\le k\le \text{log}\left(c_2\text{log}\frac{1}{1-\gamma }\right)$. We analyze (C.19) under two different cases:

1. If there exists some integer k₀ with $1\le k_0<\lceil \text{log}\left(c_2\text{log}\frac{1}{1-\gamma }\right)\rceil$, such that

   $$u_{k_0}\le \text{max}\left\{1,\frac{6\gamma \xi }{{(1-\gamma )}^2}\right\},$$
   then it is straightforward to check from (C.19) that

   $$u_{k_0+1}\le 3\text{max}\left\{30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}},\frac{2\gamma \xi }{{(1-\gamma )}^2}\right\}$$ (C.20)

   as long as T ≥ C₃(1 − γ)⁻⁴ log⁴ T log(KT/δ) for some sufficiently large constant C₃ > 0.

2. Otherwise we have $u_k>\text{max}\left\{1,\frac{6\gamma \xi }{{(1-\gamma )}^2}\right\}$ for all $1\le k<\lceil \text{log}\left(c_2\text{log}\frac{1}{1-\gamma }\right)\rceil$. This together with (C.19) suggests that

   $$\text{max}\left\{1,\frac{6\gamma \xi }{{(1-\gamma )}^2}\right\}<3\text{max}\left\{30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}},30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}}{u}_{k-1},\frac{2\gamma \xi }{{(1-\gamma )}^2}\right\},$$
   and therefore

   $$\text{max}\left\{30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}},30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}{u}_{k-1}},\frac{2\gamma \xi }{{(1-\gamma )}^2}\right\}=30\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}{u}_{k-1}}$$
   for all $1\le k\le \text{log}\left(c_2\text{log}\frac{1}{1-\gamma }\right)$. Let

   $$v_k≔90\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}{u}_{k-1}.}$$
   Then we know from (C.18) that

   $$u_k\le v_k \forall 1\le k\le \text{log}\left(c_2\text{log}\frac{1}{1-\gamma }\right).$$
   By applying the above two inequalities recursively, we know that

   $$u_k\le v_k={\left(\frac{8100{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}\right)}^{1/2}{u}_{k-1}^{1/2}\le {\left(\frac{8100{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}\right)}^{1/2}{v}_{k-1}^{1/2}\le {\left(\frac{8100{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}\right)}^{1/2+1/4}{u}_{k-2}^{1/4}\le {\left(\frac{8100{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}\right)}^{1/2+1/4}{v}_{k-2}^{1/4}\le \cdots \le {\left(\frac{8100{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}\right)}^{1-1/2^k}{u}_0^{1/2^k}\le \sqrt{\frac{8100{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}}{\left(\frac{1}{1-\gamma }\right)}^{1/2^k},$$
   where the last inequality holds as long as T ≥ C₃ log⁴ T log(KT/δ)(1 − γ)⁻⁴ for some sufficiently large constant C₃ > 0. Let $k_0=\tilde{c}\text{log}\text{log}\frac{1}{1-\gamma }$ for some properly chosen constant $\tilde{c}>0$ such that k₀ is an integer between 1 and $\text{log}\left(c_2\text{log}\frac{1}{1-\gamma }\right)$, we have

   $$u_{k_0}\le \sqrt{\frac{8100{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}}{\left(\frac{1}{1-\gamma }\right)}^{1/2^{k_0}}=O\left(\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}}\right).$$

   When T ≥ C₃ log⁴ T log(KT/δ)(1 − γ)⁻⁴ for some sufficiently large constant C₃ > 0, this implies that $u_{k_0}\le 1$, which contradicts with the preassumption that $u_k>\text{max}\left\{1,\frac{6\gamma \xi }{{(1-\gamma )}^2}\right\}$ for all $1\le k\le c_2\text{log}\frac{1}{1-\gamma }$.

Consequently, (C.20) must hold true and then the definition of u_k immediately leads to

$${\Vert {\Delta }_T\Vert }_{\infty }\le 90\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}}+\frac{6\gamma \xi }{{(1-\gamma )}^2}.$$

Then for any ε ∈ (0, 1], one has

$${\Vert {\Delta }_T\Vert }_{\infty }\le \epsilon +\frac{6\gamma \xi }{{(1-\gamma )}^2},$$

as long as

$$90\sqrt{\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^{4}T}}\le \epsilon .$$

Hence, if the total number of iterations T satisfies

$$T\ge C_3\frac{{\text{log}}^{4}T\text{log}\frac{KT}{\delta }}{{(1-\gamma )}^4{\epsilon }^2}$$

for some sufficiently large constant C₃ > 0, (4.3) would hold for Algorithm 1 with probability over 1 − δ.

Finally, we are left to justify (C.11). Recall the definition of x_i (cf. (C.10)), one has

$$\begin{gathered} \text{diag}\left(\text{Var}\left(x_i\mid V_{i-1},\cdots ,V_0\right)\right)={\gamma }^2{\left({\eta }_i^{(t)}\right)}^2\text{diag}\left(\text{Var}\left(\Lambda \left({\widehat{P}}_{\mathcal{K}}^{(t)}-P_{\mathcal{K}}\right)V_{i-1}\mid V_{i-1}\right)\right) \\ ={\gamma }^2{\left({\eta }_i^{(t)}\right)}^2\text{diag}\left(\Lambda \text{Var}\left(\left({\widehat{P}}_{\mathcal{K}}^{(i)}-P_{\mathcal{K}}\right)V_{i-1}\mid V_{i-1}\right){\Lambda }^{\top }\right) \\ ={\gamma }^2{\left({\eta }_i^{(t)}\right)}^2{\left\{\lambda {(s,a)}^2{\text{Var}}_{P_{\mathcal{K}}}\left(V_{i-1}\right)\right\}}_{s,a}, \end{gathered}$$

where the notation ${\text{Var}}_{P_{\mathcal{K}}}\left(V_{i-1}\right)$ is defined in (A.2). Plugging this into the definition of W_t leads to

$$\begin{gathered} W_t={\gamma }^2\sum _{i=(1-\alpha )t+1}^t{\left({\eta }_i^{(t)}\right)}^2{\left\{\lambda {(s,a)}^2{\text{Var}}_{P_{\mathcal{K}}}\left(V_{i-1}\right)\right\}}_{s,a} \\ ={\gamma }^2\sum _{i=(1-\alpha )t+1}^t{\left({\eta }_i^{(t)}\right)}^2{\left\{\lambda {(s,a)}^2\left(P_{\mathcal{K}}\left(V_{i-1}\circ V_{i-1}\right)-\left(P_{\mathcal{K}}{V}_{i-1}\right)\circ \left(P_{\mathcal{K}}{V}_{i-1}\right)\right)\right\}}_{s,a}. \end{gathered}$$ (C.21)

Then we introduce a useful claim as follows. The proof is deferred to Appendix C.2.

Claim 1.

For any state-action pair $(s,a)\in \mathcal{S}\times \mathcal{A}$ and vector $V\in {ℝ}^{|\mathcal{S}|}$, one has

$$\begin{gathered} \lambda {\left(s,a\right)}^2\left(P_{\mathcal{K}}(V\circ V)-\left(P_{\mathcal{K}}V\right)\circ \left(P_{\mathcal{K}}V\right)\right) \\ \le \lambda (s,a)P_{\mathcal{K}}(V\circ V)-\left(\lambda (s,a)P_{\mathcal{K}}V\right)\circ \left(\lambda (s,a)P_{\mathcal{K}}V\right). \end{gathered}$$ (C.22)

By invoking this claim with V = Vⁱ⁻¹ and taking collectively with (C.21), one has

$$\begin{gathered} W_t\le {\gamma }^2\sum _{i=(1-\beta )t+1}^t{\left({\eta }_i^{(t)}\right)}^2{\left\{\lambda (s,a)P_{\mathcal{K}}\left(V_{i-1}\circ V_{i-1}\right)-\left(\lambda (s,a)P_{\mathcal{K}}{V}_{i-1}\right)\circ \left(\lambda (s,a)P_{\mathcal{K}}{V}_{i-1}\right)\right\}}_{s,a} \\ ={\gamma }^2\sum _{i=(1-\beta )t+1}^t{\left({\eta }_i^{(t)}\right)}^2\left[\Lambda P_{\mathcal{K}}\left(V_{i-1}\circ V_{i-1}\right)-\left(\Lambda P_{\mathcal{K}}{V}_{i-1}\right)\circ \left(\Lambda P_{\mathcal{K}}{V}_{i-1}\right)\right] \\ ={\gamma }^2\sum _{i=(1-\beta )t+1}^t{\left({\eta }_i^{(t)}\right)}^2{\text{Var}}_{\overline{P}}\left(V_{i-1}\right), \end{gathered}$$

which is the desired result.

C.2. Proof of Claim 1

To simplify notations in this proof, we use ${\left[{\lambda }_i\right]}_{i=1}^K,{\left[P_{i,j}\right]}_{1\le i\le K,1\le j\le |\mathcal{S}|}$ and ${\left[V_i\right]}_{i=1}^{|\mathcal{S}|}$ to denote λ(s, a), $P_{\mathcal{K}}$ and V respectively. Then one has

$$\begin{gathered} \lambda (s,a)P_{\mathcal{K}}(V\circ V)-\left(\lambda (s,a)P_{\mathcal{K}}V\right)\circ \left(\lambda (s,a)P_{\mathcal{K}}V\right) \\ -\lambda {(s,a)}^2\left(P_{\mathcal{K}}(V\circ V)-\left(P_{\mathcal{K}}V\right)\circ \left(P_{\mathcal{K}}V\right)\right) \\ =\sum _{i=1}^K\sum _{j=1}^{\left|\mathcal{S}\right|}{\lambda }_i{P}_{i,j}{V}_j^2-{\left(\sum _{i=1}^K\sum _{j=1}^{\left|\mathcal{S}\right|}{\lambda }_i{P}_{i,j}{V}_j\right)}^2-\sum _{i=1}^K\sum _{j=1}^{\left|\mathcal{S}\right|}{\lambda }_i^2{P}_{i,j}{V}_j^2+\sum _{i=1}^K{\lambda }_i^2{\left(\sum _{j=1}^{\left|\mathcal{S}\right|}{P}_{i,j}{V}_j\right)}^2 \\ =\sum _{i=1}^K\sum _{j=1}^{\left|\mathcal{S}\right|}{\lambda }_i{P}_{i,j}{V}_j\left[\left(1-{\lambda }_i\right)V_j-\sum _{i^{\prime }\ne i}\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{\lambda }_{i^{\prime }}{P}_{i^{\prime },j^{\prime }}{V}_{j^{\prime }}\right]. \\ =\sum _{i=1}^K\sum _{j=1}^{\left|\mathcal{S}\right|}{\lambda }_i{P}_{i,j}{V}_j\left[\left(\sum _{i^{\prime }=1}^K\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{\lambda }_{i^{\prime }}{P}_{i^{\prime },j^{\prime }}-{\lambda }_i\right)V_j-\sum _{i^{\prime }\ne i}\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{\lambda }_{i^{\prime }}{P}_{i^{\prime },j^{\prime }}{V}_{j^{\prime }}\right] \\ =\sum _{i=1}^K\sum _{j=1}^{\left|\mathcal{S}\right|}\sum _{i^{\prime }\ne i}\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{\lambda }_i{P}_{i,j}{V}_j{\lambda }_{i^{\prime }}{P}_{i^{\prime },j^{\prime }}\left(V_j-V_{j^{\prime }}\right) \end{gathered}$$

where in the penultimate equality, we use the fact that

$$\sum _{i^{\prime }=1}^K\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{\lambda }_{i^{\prime }}{P}_{i^{\prime },j^{\prime }}=\lambda (s,a)P_{\mathcal{K}}1=1.$$

It follows that

$$\begin{gathered} \lambda (s,a)P_{\mathcal{K}}(V\circ V)-\left(\lambda (s,a)P_{\mathcal{K}}V\right)\circ \left(\lambda (s,a)P_{\mathcal{K}}V\right) \\ -\lambda {(s,a)}^2\left(P_{\mathcal{K}}(V\circ V)-\left(P_{\mathcal{K}}V\right)\circ \left(P_{\mathcal{K}}V\right)\right) \\ =\sum _{i=1}^K\sum _{1\le i^{\prime }<i}\sum _{j=1}^{\left|\mathcal{S}\right|}\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}\left[{\lambda }_i{P}_{i,j}{V}_j{\lambda }_{i^{\prime }}{P}_{i^{\prime },j^{\prime }}\left(V_j-V_{j^{\prime }}\right)+{\lambda }_{i^{\prime }}{P}_{i^{\prime },j}{V}_j{\lambda }_i{P}_{i,j^{\prime }}\left(V_j-V_{j^{\prime }}\right)\right] \\ =\sum _{i=1}^K\sum _{1\le i^{\prime }<i}{\lambda }_i{\lambda }_{i^{\prime }}\left[\sum _{j=1}^{\left|\mathcal{S}\right|}\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{P}_{i,j}{V}_j{P}_{i^{\prime },j^{\prime }}\left(V_j-V_{j^{\prime }}\right)+\sum _{j=1}^{\left|\mathcal{S}\right|}\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{P}_{i^{\prime },j}{V}_j{P}_{i,j^{\prime }}\left(V_j-V_{j^{\prime }}\right)\right] \\ \stackrel{\text{ (i) }}{=}\sum _{i=1}^K\sum _{1\le i^{\prime }<i}{\lambda }_i{\lambda }_{i^{\prime }}\left[\sum _{j=1}^{\left|\mathcal{S}\right|}\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{P}_{i,j}{V}_j{P}_{i^{\prime },j^{\prime }}\left(V_j-V_{j^{\prime }}\right)+\sum _{j=1}^{\left|\mathcal{S}\right|}\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{P}_{i^{\prime },j^{\prime }}{V}_{j^{\prime }}{P}_{i,j}\left(V_{j^{\prime }}-V_j\right)\right] \\ =\sum _{i=1}^K\sum _{1\le i^{\prime }<i}{\lambda }_i{\lambda }_{i^{\prime }}\left[\sum _{j=1}^{\left|\mathcal{S}\right|}\sum _{j^{\prime }=1}^{\left|\mathcal{S}\right|}{P}_{i,j}{P}_{i^{\prime },j^{\prime }}{\left(V_j-V_{j^{\prime }}\right)}^2\right] \\ \ge 0, \end{gathered}$$

where in (i), we exchange the indices j and j′.

D. Feature dimension and the number of anchor state-action pairs

The assumption that the feature dimension (denoted by K_d) and the number of anchor state-action pairs (denoted by K_n) are equal is actually non-essential. In what follows, we will show that if K_d ≠ K_n, then we can modify the current feature mapping $\varphi :\mathcal{S}\times \mathcal{A}\to {ℝ}^{K_{\text{d}}}$ to achieve a new feature mapping ${\varphi }^{\prime }:\mathcal{S}\times \mathcal{A}\to {ℝ}^{K_{\text{n}}}$ that does not change the transition model P. By doing so, the new feature dimension K_n equals to the number of anchor state-action pairs.

To begin with, we recall from Definition 1 that there exists K_d unknown functions ${\psi }_1,\cdots ,{\psi }_{K_{\text{d}}}:\mathcal{S}\to ℝ$, such that

$$P\left(s^{\prime }\mid s,a\right)=\sum _{k=1}^{K_{\text{d}}}{\varphi }_k(s,a){\psi }_k\left(s^{\prime }\right),$$

for every $(s,a)\in \mathcal{S}\times \mathcal{A}$ and $s^{\prime }\in \mathcal{S}$. In addition, we also recall from Assumption 1 that there exists $\mathcal{K}\subseteq \mathcal{S}\times \mathcal{A}$ with $|\mathcal{K}|=K_{\text{n}}$ such that for any $(s,a)\in \mathcal{S}\times \mathcal{A}$,

$$\varphi (s,a)=\sum _{i:\left(s_i,a_i\right)\in \mathcal{K}}{\lambda }_i(s,a)\varphi \left(s_i,a_i\right)\in {ℝ}^{K_{\text{d}}} for \sum _{i=1}^{K_{\text{n}}}{\lambda }_i(s,a)=1 and {\lambda }_i(s,a)\ge 0.$$

Case 1:

K_d > K_n. In this case, the vectors in $\{\varphi (s,a):(s,a)\in \mathcal{K}\}$ are linearly independent. For ease of presentation and without loss of generality, we assume that K_d = K_n + 1. This indicates that the matrix $\Phi \in {ℝ}^{K_{\text{d}}\times (|\mathcal{S}||\mathcal{A}|)}$ whose columns are composed of the feature vectors of all state-action pairs has rank K_n and is hence not full row rank. This suggests that there exists K_n linearly independent rows (without loss of generality, we assume they are the first K_n rows). We can remove the last row from Φ to obtain ${\Phi }^{\prime }≔{\Phi }_{1:K_{\text{n}},:}\in {ℝ}^{K_{\text{n}}\times (|\mathcal{S}||\mathcal{A}|)}$ such that Φ′ is full row rank. Then we show that we can actually use the columns of Φ′ as new feature mappings. To see why this is true, note that the last row ${\Phi }_{K_{\text{n}}+1,:}$ can be represented as a linear combination of the first K_n rows, namely there must exist constants ${\left\{c_k\right\}}_{k=1}^{K_{\text{n}}}$ such that for any $(s,a)\in \mathcal{S}\times \mathcal{A}$,

$${\varphi }_{K_{\text{n}}+1}(s,a)=\sum _{k=1}^{K_{\text{n}}}{c}_k{\varphi }_k(s,a).$$

Define ${\psi }_k^{\prime }={\psi }_k+c_k{\psi }_{K_{\text{n}}+1}$ for k = 1, …, K_n, we have

$$\begin{gathered} P\left(s^{\prime }\mid s,a\right)=\sum _{k=1}^{K_{\text{d}}}{\varphi }_k(s,a){\psi }_k\left(s^{\prime }\right)={\varphi }_{K_{\text{n}}+1}(s,a){\psi }_{K_{\text{n}}+1}\left(s^{\prime }\right)+\sum _{k=1}^{K_{\text{n}}}{\varphi }_k(s,a){\psi }_k\left(s^{\prime }\right) \\ =\sum _{k=1}^{K_{\text{n}}}{\varphi }_k(s,a)\left[{\psi }_k\left(s^{\prime }\right)+c_k{\psi }_{K_{\text{n}}+1}\left(s^{\prime }\right)\right]=\sum _{k=1}^{K_{\text{n}}}{\varphi }_k(s,a){\psi }_k^{\prime }\left(s^{\prime }\right), \end{gathered}$$

which is linear with respect to the new K_n dimensional feature vectors. It is also straightforward to check that the new feature mapping satisfies Assumption 1 with the original anchor state-action pairs K.

Case 2:

K_d < K_n. For ease of presentation and without loss of generality, we assume that K_n = K_d + 1 and that the subspace spanned by the feature vectors of anchor state-action pairs is non-degenerate, i.e., has rank K_d (otherwise we can use similar method as in Case 1 to further reduce the feature dimension K_d). In this case, the matrix ${\Phi }_{\mathcal{K}}\in {ℝ}^{K_{\text{d}}\times K_{\text{n}}}$ whose columns are composed of the feature vectors of anchor state-action pairs has rank K_d. We can add K_n − K_d = 1 new row to ${\Phi }_{\mathcal{K}}$ to obtain ${\Phi }_{\mathcal{K}}^{\prime }\in {ℝ}^{K_{\text{n}}\times K_{\text{n}}}$ such that ${\Phi }_{\mathcal{K}}^{\prime }$ has full rank K_n. Then we let the columns of ${\Phi }_{\mathcal{K}}^{\prime }={\left[{\varphi }^{\prime }(s,a)\right]}_{(s,a)\in \mathcal{K}}$ to be the new feature vectors of the anchor state-action pairs, and define the new feature vectors for all other state-action pairs $(s,a)\notin \mathcal{K}$ by

$${\varphi }^{\prime }(s,a)=\sum _{i:\left(s_i,a_i\right)\in \mathcal{K}}{\lambda }_i(s,a){\varphi }^{\prime }\left(s_i,a_i\right).$$

We can check that the transition model P is not changed if we let ${\psi }_{K_{\text{n}}}\left(s^{\prime }\right)=0$ for every $s^{\prime }\in \mathcal{S}$. It is also straightforward to check that Assumption 1 is satisfied.

To conclude, when K_d ≠ K_n, we can always construct a new set of feature mappings with dimension K_n such that: (i) the feature dimension equals to the number of anchor state-action pairs (they are both K_n); (ii) the transition model can still be linearly parameterized by this new set of feature mappings; and (iii) the anchor state-action pair assumption (Assumption 1) is satisfied with the original anchor state-action pairs.