A Generalization Error for Q-Learning

Abstract

Planning problems that involve learning a policy from a single training set of finite horizon trajectories arise in both social science and medical fields. We consider Q-learning with function approximation for this setting and derive an upper bound on the generalization error. This upper bound is in terms of quantities minimized by a Q-learning algorithm, the complexity of the approximation space and an approximation term due to the mismatch between Q-learning and the goal of learning a policy that maximizes the value function.

1. Introduction

In many areas of the medical and social sciences the following planning problem arises. A training set or batch of n trajectories of T +1-decision epochs is available for estimating a policy. A decision epoch at time t, t = 0,1,...,T , is composed of information observed at time t, O_t , an action taken at time t, A_t and a reward, R_t . For example there are currently a number of ongoing large clinical trials for chronic disorders in which, each time an individual relapses, the individual is re-randomized to one of several further treatments (Schneider et al., 2001; Fava et al., 2003; Thall et al., 2000). These are finite horizon problems with T generally quite small, T = 2–4, with known exploration policy. Scientists want to estimate the best “strategies,” i.e. policies, for managing the disorder. Alternately the training set of n trajectories may be historical; for example data in which clinicians and their patients are followed with i! nformation about disease process, treatment burden and treatment decisions recorded through time. Again the goal is to estimate the best policy for managing the disease. Alternately, consider either catalog merchandizing or charitable solicitation; information about the client, and whether or not a solicitation is made and/or the form of the solicitation is recorded through time (Simester, Peng and Tsitsiklis, 2003). The goal is to estimate the best policy for deciding which clients should receive a mailing and the form of the mailing. These latter planning problems can be viewed as infinite horizon problems but only T decision epochs per client are recorded. If T is large, the rewards are bounded and the dynamics are stationary Markovian then this finite horizon problem provides an approximation to the discounted infinite horizon problem (Kearns, Mansour and Ng, 2000).

These planning problems are characterized by unknown system dynamics and thus can also be viewed as learning problems as well. Note there is no access to a generative model nor an online simulation model nor the ability to conduct offline simulation. All that is available is the n trajectories of T + 1 decision epochs. One approach to learning a policy in this setting is Q-learning (Watkins, 1989) since the actions in the training set are chosen according to a (non-optimal) exploration policy; Q-learning is an off-policy method (Sutton and Barto, 1998). When the observables are vectors of continuous variables or are otherwise of high dimension, Q-learning must be combined with function approximation.

The contributions of this paper are as follows. First a version of Q-learning with function approximation, suitable for learning a policy with one training set of finite horizon trajectories and a large observation space, is introduced; this “batch” version of Q-learning processes the entire training set of trajectories prior to updating the approximations to the Q-functions. An incremental implementation of batch Q-learning results in one-step Q-learning with function approximation. Second performance guarantees for this version of Q-learning are provided. These performance guarantees do not assume assume that the system dynamics are Markovian. The performance guarantees are upper bounds on the average difference in value functions or more specifically the average generalization error. Here the generalization error for batch Q-learning is defined analogous to the generalization error in supervised learning (Schapire et al., 1998); it is the average diffe! rence in value when using the optimal policy as compared to using the greedy policy (from Q-learning) in generating a separate test set. The performance guarantees are analogous to performance guarantees available in supervised learning (Anthony and Bartlett, 1999).

The upper bounds on the average generalization error permit an additional contribution. These upper bounds illuminate the mismatch between Q-learning with function approximation and the goal of finding a policy maximizing the value function (see the remark following Lemma 2 and the third remark following Theorem 2). This mismatch occurs because the Q-learning algorithm with function approximation does not directly maximize the value function but rather this algorithm approximates the optimal Q-function within the constraints of the approximation space in a least squares sense; this point is discussed as some length in section 3 of Tsitsiklis and van Roy (1997).

In the process of providing an upper bound on the average generalization error, finite sample bounds on the difference in average values resulting from different policies are derived. There are three terms in the upper bounds. The first term is a function of the optimization criterion used in batch Q-learning, the second term is due to the complexity of the approximation space and the last term is an approximation error due to the above mentioned mismatch. The third term which is a function of the complexity of the approximation space is similar in form to generalization error bounds derived for supervised learning with neural networks as in Anthony and Bartlett (1999). From the work of Kearns, Mansour, and Ng (1999, 2000) and Peshkin and Shelton (2002), we expect and find as well here that the number of trajectories needed to guarantee a specified error level is exponential in the horizon time, T . The upper bound does not depend on the dimension of the observables O_t ’s. This is in contrast to the results of Fiechter (1994 Fiechter (1997) in which the upper bound on the average generalization error depends on the number of possible values for the observables.

A further contribution is that the upper bound on the average generalization error provides a mechanism for generalizing ideas from supervised learning to reinforcement learning. For example if the optimal Q-function belongs to the approximation space, then the upper bounds imply that batch Q-learning is a PAC reinforcement learning algorithm as in Feichter (1994 in Feichter (1997); see the first remark following Theorem 1. And second the upper bounds provide a starting point in using structural risk minimization for model selection (see the second remark after Theorem 1).

In Section 2, we review the definition of the value function and Q-function for a (possibly non-stationary, non-Markovian) finite horizon decision process. Next we review batch Q-learning with function approximation when the learning algorithm must use a training set of n trajectories. In Section 5 we provide the two main results, both of which provide the number of trajectories needed to achieve a given error level with a specified level of certainty.

2. Preliminaries

In the following we use upper case letters, such as O and A, to denote random variables and lower case letters, such as o and a, to denote instantiates or values of the random variables. Each of the n trajectories is composed of the sequence {O₀, A₀, O₁, . . . , A_T , O_{T +1}} where T is a finite constant. Define O_t = {O₀,..., O_t } and similarly for A_t . Each action A_t takes values in finite, discrete action space A and O_t takes values in the observation space O. The observation space may be multidimensional and continuous. The arguments below will not require the Markovian assumption with the value of O_t equal to the state at time t. The rewards are R_t = r_t (O_t , A_t , O_t+1) for r_t a reward function and for each 0 ≤ t ≤ T (if the Markov assumption holds then replace O_t with O_t and A_t with A_t ). We assume that the rewards are bounded, taking values in the interval [0,1].

We assume the trajectories are sampled at random according to a fixed distribution denoted by P. Thus the trajectories are generated by one fixed distribution. This distribution is composed of the unknown distribution of each O_t conditional on (O_t−1, A_t−1) (call these unknown conditional densities {f₀,... f_T }) and an exploration policy for generating the actions. Denote the exploration policy by p_T = {p₀,..., p_T} where the probability that action a is taken given history {O_t , A_t−1} is p_t (a|O_t , A_t−1) (if the Markov assumption holds then, as before, replace O_t with O_t and A_t−1 with A_t−1.) We assume that p_t (a|o_t , a_t−1) > 0 for each action a ∈ A and for each possible value (o_t , a_t−1); that is, at each time all actions are possible. Then the likelihood (under P) of the trajectory, {o₀, a₀, o₁,..., a_T , o_{T +1}} is

$$f_0(o_0)p_0(a_0 | o_0)\prod _{t=1}^T{f}_t(o_t | o_{t-1},a_{t-1})p_t(a_t | o_t,a_{t-1})f_{T+1}(o_{T+1} | o_T,a_T).$$ (1)

Denote expectations with respect to the distribution P by an E.

Define a deterministic, but possibly non-stationary and non-Markovian, policy, π, as a sequence of decision rules, {π₁,..., π_T }, where the output of the time t decision rule, π_t (o_t , a_t−1), is an action. Let the distribution P_π denote the distribution of a trajectory whereby the policy π is used to generate the actions. Then the likelihood (under P_π) of the trajectory {o₀, a₀, o₁,..., a_T , o_{T +1}} is

$$f_0(o_0)1_{a_0={\pi }_0(o_0)}\prod _{j=1}^T{f}_j(o_j | o_{j-1},a_{j-1})1_{a_j={\pi }_j( o_j,a_{j-1})}{f}_{T+1}(o_{T+1} | o_T,a_T)$$ (2)

where for a predicate W , 1_W is 1 if W is true and is 0 otherwise. Denote expectations with respect to the distribution P_π by an E_π.

Note that since (1) and (2) differ only in regard to the policy for generating actions, an expectation with respect to either P or P_π that does not involve integration over the policy results in the same quantity. For example, E [R_t |O_t , A_t ] = E_π [R_t |O_t , A_t ], for any policy π.

Let Π be the collection of all policies. In a finite horizon planning problem (permitting non-stationary, non-Markovian policies) the goal is to estimate a policy that maximizes $E_{\pi }[{\sum }_{j=1}^T{R}_j | O_0=o_0]$ over π ∈ Π. If the system dynamics are Markovian and each r _j(o _j, a _j, o _j+1) = γj_r(o _j, a _j, o _j+1) for r a bounded reward function and γ ∈ (0,1) a discount factor, then this finite horizon problem provides an approximation to the discounted infinite horizon problem (Kearns Mansour and Ng, 2000) for T large.

Given a policy, π, the value function for an observation, o₀, is

$$V_{\pi }(o_0)=E_{\pi }\left[\sum _{j=1}^T{R}_j | O_0=o_0\right].$$

The t-value function for policy π is the value of the rewards summed from time t on and is

$$V_{\pi ,t}(o_t,a_{t-1})=E_{\pi }\left[\sum _{j=t}^T{R}_j |O_t=o_t,A_{t-1}=\hspace{0.17em}{a}_{t-1}\right].$$

If the Markovian assumption holds then (o_t , a_t−1) in the definition of V_π,t is replaced by o_t . Note that the time 0 value function is simply the value function (V_π,0 = V_π). For convenience, set V_{π,T +1} = 0. Then the value functions satisfy the following relationship:

$$V_{\pi ,t}(o_t,a_{t-1})=E_{\pi }[R_t+V_{\pi ,t+1}(O_{t+1},A_t)|O_t=o_t,A_{t-1}=a_{t-1}]$$

for t = 0,...,T . The time t Q-function for policy π is

$$Q_{\pi ,t}(o_t,a_t)=E[R_t+V_{\pi ,t+1}(O_{t+1},A_t)|O_t=o_t,A_t=a_t].$$

(The subscript, π, can be omitted as this expectation is with respect to the distribution of O_t+1 given (O_t , A_t ), f_t+1; this conditional distribution does not depend on the policy.) In Section 4 we express the difference in value functions for policy $\tilde{\pi }$ and policy π in terms of the advantages (as defined in Baird, 1993). The time t advantage is

$${\mu }_{\pi ,t}(o_t,a_t)=Q_{\pi ,t}(o_t,a_t)-V_{\pi ,t}(o_t,a_{t-1}).$$

The advantage can be interpreted as the gain in performance obtained by following action a_t at time t and thereafter policy π as compared to following policy π from time t on.

The optimal value function V ^*(o) for an observation o is

$$V^{*}(o)=\underset{\pi \in \mathrm{\Pi }}{max}{V}_{\pi }(o)$$

and the optimal t-value function for history (o_t , a_t−1) is

$$V_t^{*}(o_t,a_{t-1})=\underset{\pi \in \mathrm{\Pi }}{max}{V}_{\pi ,t}(o_t,a_{t-1}).$$

As is well-known, the optimal value functions satisfy the Bellman equations (Bellman, 1957)

$$V_t^{*}(o_t,a_{t-1})=\underset{a_t\in A}{max}E[R_t+V_{t+1}^{*}(O_{t+1},A_t) | O_t=o_t,A_t=a_t].$$

Optimal, deterministic, time t decision rules must satisfy

$${\pi }_t^{*}(o_t,a_{t-1})\in arg\underset{a_t\in A}{max}E[R_t+V_{t+1}^{*}(O_{t+1},A_t) | O_t=o_t,A_t=a_t].$$

The optimal time t Q-function is

$$Q_t^{*}(o_t,a_t)=E[R_t+V_{t+1}^{*}(O_{t+1},A_t)|O_t=o_t,A_t=a_t],$$

and thus the optimal time t advantage, which is given by

$${\mu }_t^{*}(o_t,a_t)=Q_t^{*}(o_t,a_t)-V_t^{*}(o_t,a_{t-1}),$$

is always nonpositive and furthermore it is maximized in a_t at a_t = π_t^*(o_t , a_t−1).

3. Batch Q-Learning

We consider a version of Q-learning for use in learning a non-stationary, non-Markovian policy with one training set of finite horizon trajectories. The term “batch” Q-learning is used to emphasize that learning occurs only after the collection of the training set. The Q-functions are estimated using an approximator (i.e. neural networks, decision-trees etc.) (Bertsekas and Tsitsiklis, 1996; Tsitsiklis and van Roy, 1997) and then the estimated decision rules are the argmax of the estimated Q functions. Let Q_t be the approximation space for the tth Q-function, e.g. Q_t = {Q_t (o_t , a_t ; θ) : θ ∈θ}; θ is a vector of parameters taking values in a parameter space θ which is a subset of a Euclidean space. For convenience set Q_{T +1} equal to zero and write E_n f for the expectation of an arbitrary function, f , of a trajectory with respect to the probability obtained by choosing a trajectory uniformly from the training set of n trajectories (for example, $E_n[f(O_t)]=1/n{\sum }_{i=1}^{n}f(O_{it})$ for O_it the tth observation in the ith trajectory). In batch Q-learning using dynamic programming and function approximation solve the following backwards through time t =!T,T − 1,...,1 to obtain

$${\theta }_t\in arg\underset{\theta }{min}{E}_n{\left[R_t+\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1};{\theta }_{t+1})-Q_t(O_t,A_t;\theta )\right]}^2.$$ (3)

Suppose that Q-functions are approximated by linear combinations of p features (Q_t = {θ^T q_t (o_t , a_t ) : θ ∈ R^p}) then to achieve (3) solve backwards through time, t = T,T −1,...,0,

$$0=E_n\left[\left(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1};{\theta }_{t+1})-Q_t(O_t,A_t;{\theta }_t)\right)q_t{(O_t,A_t)}^T\right]$$ (4)

for θ_t

An incremental implementation with updates between trajectories of (3) and (4) results in one-step Q-learning (Sutton and Barto, 1998, pg. 148, put γ = 1, assume the Markov property and no need for function approximation). This is not surprising as Q-learning can be viewed as approximating least squares value iteration (Tsitsiklis and van Roy, 1996). To see the connection consider the following generic derivation of an incremental update. Denote the ith example in a training set by X_i. Define ${\widehat{\theta }}^n$ to be a solution of ${\sum }_{i=1}^{n}f(X_i,\theta )=0$ for f a given p dimensional vector of functions and each integer n. Using a Taylor series, expand ${\sum }_{i=1}^{n+1}f(X_i,{\widehat{\theta }}^{(n+1)})\mathrm{\hspace{0.17em} }\text{in\hspace{0.28em}}{\widehat{\theta }}^{(n+1)}$ about ${\widehat{\theta }}^{(n)}$ to obtain a between-example update to ${\widehat{\theta }}^{(n)}$ :

$${\widehat{\theta }}^{(n+1)}\leftarrow {\widehat{\theta }}^{(n)}+\frac{1}{n+1}{\left(E_{n+1}\left(-\frac{\partial f(X,{\widehat{\theta }}^n)}{\partial {\widehat{\theta }}^n}\right)\right)}^{-1}f(X_{n+1},{\widehat{\theta }}^n).$$

Replace $\frac{1}{n+1}{\left(E_{n+1}\left(-\frac{\partial f(X,{\widehat{\theta }}^n)}{\partial {\widehat{\theta }}^n}\right)\right)}^{-1}$ by a step-size α_n (α_n → 0 as n → ∞) to obtain a general formula for the incremental implementation. Now consider an incremental implementation of (4) for each t = 0,...,T . Then for each t, X = (O_t+1, A_t ), θ = θ_t and

$$f(X,{\theta }_t)=\left(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1};{\widehat{\theta }}_{t+1}^{(n+1)})-Q_t(O_t,A_t;{\theta }_t)\right)q_t{(O_t,A_t)}^T$$

is a vector of dimension p. The incremental update is

$${\widehat{\theta }}_t^{(n+1)}\leftarrow {\widehat{\theta }}_t^{(n)}+{\alpha }_n\left(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1};{\widehat{\theta }}_{t+1}^{(n+1)})-Q_t(O_t,A_t;{\widehat{\theta }}_t^{(n)})\right)q_t{(O_t,A_t)}^T$$

for t = 0,...,T . This is the one-step update of Sutton and Barto (1998, pg. 148) with γ = 1 and generalized to permit function approximation and nonstationary Q-functions and is analogous to the TD(0) update of Tsitsiklis and van Roy (1997) permitting non-Markovian, nonstationary value functions.

Denote the estimator of the optimal Q-functions based on the training data by ${\widehat{Q}}_t$ for t = 0,...,T (for simplicity, is omitted). The estimated policy, $\widehat{\pi }$, satisfies ${\widehat{\pi }}_t(o_t,a_{t-1})\in \hspace{0.17em}arg\mathrm{\hspace{0.17em} }{max}_{a_t}{\widehat{Q}}_t(o_t,a_t)$ for each t. Note that members of the approximation space Q_t need not be “Q-functions” for any policy. For example the Q-functions corresponding to the use of a policy π (Q_π,t, t = 0,...,T) must satisfy

$$E[R_t+V_{\pi ,t+1}(O_{t+1},A_t)|O_t,A_t]=Q_{\pi ,t}(O_t,A_t)$$

where V_π,t+1(O_t+1, A_t ) = Q_π,t+1(O_t+1, A_t , a_t+1) with a_t+1 set equal to π_t+1(O_t+1, A_t ). Q-learning does not impose this restriction on $\{{\widehat{Q}}_t,t=0,\dots ,T\}$; indeed it may be that no member of the approximation space can satisfy this restriction. None-the-less we refer to the ${\widehat{Q}}_t$’s as estimated Q-functions. Note also that the approximation for the Q-functions combined with the definition of the estimated decision rules as the argmax of the estimated Q functions places implicit restrictions on the set of policies that will be considered. In effect the space of interesting polices is no longer Π but rather ΠQ = {π_θ, θ ∈ θ} where π_θ = {π_1,θ,..., π_T,θ} and where each π_t,θ(o_t , a_t−1) ∈ arg max_at Q_t(o_t , a_t ; θ)for some Q_t ∈ Q_t.

4. Generalization Error

Define the generalization error of a policy π at an observation o₀ as the average difference between the optimal value function and the value function resulting from the use of policy π in generating a separate test set. The generalization error of policy π at observation o₀ can be written as

$$V^{*}(o_0)-V_{\pi }(o_0)=-E_{\pi }\left[\sum _{t=0}^T{\mu }_t^{*}(O_t,A_t)|O_0=o_0\right]$$ (5)

where E_π denotes the expectation using the likelihood (2). So the generalization error can be expressed in terms of the optimal advantages evaluated at actions determined by policy π; that is when each A_t = π_t (O_t , A_t−1). Thus the closer each optimal advantage, μt *(O_t , A_t ) for A_t = π_t (O_t , A_t−1) is to zero, the smaller the generalization error. Recall that the optimal advantage, μt *(O_t , A_t ), is zero when A_t = π_t *(O_t , A_t−1). The display in (5) follows from Kakade’s (ch. 5, 2003) expression for the difference between the value functions for two policies.

Lemma 1

Given policies $\tilde{\pi }$ and π,

$$V_{\tilde{\pi }}(o_0)-V_{\pi }(o_0)=-E_{\pi }\left[\sum _{t=0}^T{\mu }_{\tilde{\pi },t}(O_t,A_t)|O_0=o_0\right].$$

Set $\tilde{\pi }$ = π* to obtain (5). An alternate to Kakade’s (2003) proof is as follows.

Proof. First note

$$V_{\pi }(o_0)=E_{\pi }\left[\sum _{t=0}^T{R}_t|O_0=o_0\right]=E_{\pi }\left[E_{\pi }\left[\sum _{t=0}^T{R}_t|O_T,A_T\right]|O_0=o_0\right].$$ (6)

And $E_{\pi }\left[{\sum }_{t=0}^T{R}_t|O_T,A_T\right]$ is the expectation with respect to the distribution of O_{T + 1} given the history (O_T, A_T); this is the density f_{T + 1} from Section 2 and f_{T + 1} is independent of the policy used to choose the actions. Thus we may subscript E by either π or $\tilde{\pi }$ without changing the expectation; $E_{\pi }\left[{\sum }_{t=0}^T{R}_t|O_T,A_T\right]=E_{\tilde{\pi }}\left[{\sum }_{t=0}^T{R}_t|O_T,A_T\right]={\sum }_{t=0}^{T-1}{R}_t+Q_{\tilde{\pi },T}(O_T,A_T).$ The conditional expectation can be written in a telescoping sum as

$$\begin{array}{l}{E}_{\pi }\left[\sum _{t=0}^T{R}_t|O_T,A_T\right]=\sum _{t=0}^T{Q}_{\tilde{\pi },t}(O_t,A_t)-\hspace{0.17em}{V}_{\tilde{\pi },t}(O_t,A_{t-1})\\ +\hspace{0.17em}\sum _{t=1}^T{R}_{t-1}+\hspace{0.17em}{V}_{\tilde{\pi },t}(O_t,A_{t-1})-Q_{\tilde{\pi },t-1}(O_{t-1},A_{t-1})\\ +\hspace{0.17em}{V}_{\tilde{\pi },0}(O_0)\end{array}$$

The first sum is the sum of the advantages. The second sum is a sum of temporal-difference errors; integrating the temporal-difference error with respect to the conditional distribution of O_t given (O_{t − 1}, A_{t − 1}), denoted by f_t in Section 2, we obtain zero,

$$E\hspace{0.17em}[R_{t-1}+V_{\tilde{\pi },t}(O_t,A_{t-1})|O_{t-1},A_{t-1}]=Q_{\tilde{\pi },t-1}(O_{t-1},A_{t-1})$$

(as before E denotes expectation with respect to (1); recall that expectations that do not integrate over the policy can be written either with an E or an E_π). Substitute the telescoping sum into (6) and note that $V_{\tilde{\pi },0}(O_0)=V_{\tilde{\pi }}(O_0)$ to obtain the result. ▪

In the following Lemma the difference between value functions corresponding to two policies, $\tilde{\pi }$ and π, is expressed in terms of both the L₁ and L₂ distances between the optimal Q-functions and any functions {Q₀, Q₁,..., Q_T} satisfying π_t (o_t, a_{t − 1}) ∈ arg max_at Q_t (o_t, a_t), t = 0,...,T and any functions $\left\{{\tilde{Q}}_0,{\tilde{Q}}_1,\dots ,{\tilde{Q}}_T\right\}$ satisfying ${\tilde{\pi }}_t(o_t,a_{t-1})\hspace{0.17em}\in \hspace{0.17em}arg\hspace{0.17em}{max}_{a_t}{\tilde{Q}}_t(o_t,a_t)$, t = 0,...,T. We assume that there exists a positive constant, L for which p_t (a_t|o_t, a_{t − 1}) ≥ L⁻¹ for each t and all pairs (o_t, a_{t − 1}); if the stochastic decision rule, p_t, were uniform then L would be the size of the action space.

Lemma 2

For all functions, Q_t satisfying π_t(o_t, a_{t − 1}) ∈ arg max_at Q_t(o_t, a_t), t = 0,...,T, and all functions ${\tilde{Q}}_t$ satisfying ${\tilde{\pi }}_t(o_t,a_{t-1})\hspace{0.17em}\in \hspace{0.17em}arg\hspace{0.17em}{max}_{a_t}{\tilde{Q}}_t(o_t,a_t)$, t = 0,...,T we have,

$$\begin{array}{l}\left|V_{\tilde{\pi }}(o_0)-V_{\pi }(o_0)\right|\hspace{0.17em}\le \hspace{0.17em}\sum _{t=0}^{T}2L^{t+1}E\left[\left|Q_t(O_t,A_t)-{\tilde{Q}}_t(O_t,A_t)\right|O_0=o_0\right]\\ +\hspace{0.17em}\hspace{0.17em}\sum _{t=0}^{T}2L^{t+1}E\left[\left|{\tilde{Q}}_t(O_t,A_t)-Q_{\tilde{\pi },t}(O_t,A_t)\right|O_0=o\right]\end{array}$$

and

$$\begin{array}{l}\left|V_{\tilde{\pi }}(o_0)-V_{\pi }(o_0)\right|\hspace{0.17em}\le \hspace{0.17em}\sum _{t=0}^{T}2L^{(t+1)/2}\sqrt{E\left[{(Q_t(O_t,A_t)-{\tilde{Q}}_t(O_t,A_t))}^2|O_0=o_0\right]}\\ +\hspace{0.17em}\sum _{t=0}^{T}2L^{(t+1)/2}\sqrt{E\left[{({\tilde{Q}}_t(O_t,A_t)-Q_{\tilde{\pi },}{}_t(O_t,A_t))}^2|O_0=o_0\right]},\end{array}$$

where E denotes expectation with respect to the distribution generating the training sample (1).

Remark:

1. Note that in general arg max_at $Q_{\tilde{\pi },}{}_t(o_t,a_t)$ may not be ${\tilde{\pi }}_t$ thus we can not choose ${\tilde{Q}}_t=Q_{\tilde{\pi },t}$. However if $\tilde{\pi }={\pi }^{*}$ then we can choose ${\tilde{Q}}_t=Q_t^{*}(=Q_{\pi *,t}\hspace{0.17em}\text{by\hspace{0.17em}definition)}$ and the second term in both upper bounds is equal to zero.

2. This result can be used to emphasize one aspect of the mismatch between estimating the optimal Q function and the goal of learning a policy that maximizes the value function. Suppose ${\tilde{Q}}_t=Q_t^{*},\tilde{\pi }=\pi *$. The generalization error is

$$V^{*}(o_0)-V_{\pi }(o_0)\le \sum _{t=0}^{T}2L^{(t+1)/2}\sqrt{E\left[{(Q_t(O_t,A_t)-Q_t^{*}(O_t,A_t))}^2|O_0=o_0\right]}$$

for Q_t any function satisfying π_t(o_t, a_{t − 1}) ∈ arg max_at Q_t(o_t, a_t). Absent restrictions on the Q_t s, this inequality cannot be improved in the sense that choosing each Q_t = Q_t^* and π_t = π_t^* yields 0 on both sides of the inequality. However an inequality in the opposite direction is not possible, since as was seen in Lemma 1, V^*(o₀) − V_π(o₀) involves the Q functions only through the advantages (see also (7) below with $\tilde{\pi }={\pi }^{*}$). Thus for the difference in value functions to be small, the average difference between Q_t(o_t, a_t) − max _at Q_t(o_t, a_t) and Q^*_t(o_t, a_t) − max _at Q_t^*(o_t, a_t) must be small; this does not require that the average difference between Q_t(o_t, a_t) and Q_t^*(o_t, a_t) is small. The mismatch is not unexpected. For example, Baxter and Bartlett (2001) provide an example in which the approximation space for the value function includes a value function for which the greedy policy is optimal, yet the greedy policy found by temporal difference learning (TD(1!)) function performs very poorly.

Proof

Define μ_t(o_t, a_t) = Q_t(o_t, a_t) − max _at Q_t(o_t, a_t) for each t; note that μ_t(o_t, a_t) evaluated at a_t = π_t(o_t, a_{t − 1}) is zero. Start with the result of Lemma 1. Then note the difference between the value functions can be expressed as

$$V_{\tilde{\pi }}(o_0)-V_{\pi }(o_0)=\sum _{t=0}^T{E}_{\pi }\left[{\mu }_t(O_t,A_t)-{\mu }_{\tilde{\pi },t}(O_t,A_t)|O_0=o_0\right].$$ (7)

since P_π puts a_t = π_t(o_t, a_{t − 1}) and μ_t (o_t, a_t) = 0 for this value of a_t. When it is clear from the context μ_t $({\mu }_{\tilde{\pi },t})$ is used as abbreviation for μ_t (O_t, A_t) $({\mu }_{\tilde{\pi },t}(O_t,A_t))$ in the following. Also $Q_{\tilde{\pi },t}$(O_t, A_{t − 1}, a_t) with a_t replaced by ${\tilde{\pi }}_t$ (O_t, A_{t − 1}) is written as $Q_{\tilde{\pi },t}(O_t,A_{t-1},{\tilde{\pi }}_t).$ Consider the absolute value of the tth integrand in (7):

$$\begin{array}{l}\left|{\mu }_t-{\mu }_{\tilde{\pi },t}\right|\\ =\left|Q_t(O_t,A_t)-\underset{a_t}{max}{Q}_t(O_t,A_{t-1},a_t)-Q_{\tilde{\pi },t}(O_t,A_t)+Q_{\tilde{\pi },t}(O_t,A_{t-1},{\tilde{\pi }}_t)\right|\\ \le \left|Q_t(O_t,A_t)-Q_{\tilde{\pi },t}(O_t,A_t)\right|+\left|\underset{a_t}{max}{Q}_t(O_t,A_{t-1},a_t)-Q_{\tilde{\pi },t}(O_t,A_{t-1},{\tilde{\pi }}_t)\right|.\end{array}$$

Since max_at ${\tilde{Q}}_t(O_t,A_{t-1},a_t)-{\tilde{Q}}_t(O_t,A_{t-1},{\tilde{\pi }}_t)$and for any functions, h and h′, |max _at h(a_t) − max_at h′(a_t)|≤ max_at |h(a_t) − h′ (a_t)|,

$$\begin{array}{l}\left|\underset{a_t}{max}{Q}_t(O_t,A_{t-1},a_t)-Q_{\tilde{\pi },t}(O_t,A_{t-1},{\tilde{\pi }}_t)\right|\\ \le \underset{a_t}{max}\left|Q_t(O_t,A_{t-1},a_t)-{\tilde{Q}}_t(O_t,A_{t-1},a_t)\right|\\ +\left|{\tilde{Q}}_t(O_t,A_{t-1},{\tilde{\pi }}_t)-Q_{\tilde{\pi },t}(O_t,A_{t-1},{\tilde{\pi }}_t)\right|.\end{array}$$

We obtain $\left|{\mu }_t-{\mu }_{\tilde{\pi },t}\right|$

$$\begin{array}{l}\le \underset{a_t}{2max}\left|Q_t(O_t,A_{t-1},a_t)-{\tilde{Q}}_t(O_t,A_{t-1},a_t)\right|+\underset{a_t}{2max}\left|{\tilde{Q}}_t(O_t,A_{t-1},a_t)-{\tilde{Q}}_{\tilde{\pi },t}(O_t,A_{t-1},a_t)\right|\\ \le 2L\sum _{a_t}\left|Q_t(O_t,A_{t-1},a_t)-{\tilde{Q}}_t(O_t,A_{t-1},a_t)\right|p_t(a_t|O_t,A_{t-1})\\ +2L\sum _{a_t}\left|{\tilde{Q}}_t(O_t,A_{t-1},a_t)-Q_{\tilde{\pi },t}(O_t,A_{t-1},a_t)\right|p_t(a_t|O_t,A_{t-1}).\end{array}$$ (8)

Insert the above into (7) and use Lemma A1 to obtain $\left|V_{\tilde{\pi }}(o_0)-V_{\pi }(o_0)\right|$

$$\begin{array}{l}\le 2L\sum _{t=0}^T{E}_{\pi }\left[\sum _{a_t}\left|{\tilde{Q}}_t(O_t,A_{t-1},a_t)-{\tilde{Q}}_t(O_t,A_{t-1},a_t)\right|p_t(a_t\left|O_t,A_{t-1})\right|O_0=o_0\right]\\ +E_{\pi }\left[\sum _{a_t}\left|{\tilde{Q}}_t(O_t,A_{t-1},a_t)-Q_{\tilde{\pi },}{}_t(O_t,A_{t-1},a_t)\right|p_t(a_t\left|O_t,A_{t-1})\right|O_0=o_0\right]\\ =2L\sum _{t=0}^{T}E\left[\left(\prod _{\ell =0}^{t-1}\frac{1_{A_{\ell }={\pi }_{\ell }}}{p\ell (A\ell |O_{\ell },A_{\ell -1})}\right)\left|Q_t-{\tilde{Q}}_t\right||O_0=o_0\right]\\ +E\left[\left(\prod _{\ell =0}^{t-1}\frac{1_{A_{\ell }={\pi }_{\ell }}}{p\ell (A\ell |O_{\ell },A_{\ell -1})}\right)\left|{\tilde{Q}}_t-Q_{\tilde{\pi },t}\right||O_0=o_0\right]\\ \le 2\sum _{t=0}^T{L}^{t+1}E\left[\left|Q_t-{\tilde{Q}}_t\right||O_0=o_0\right]+2\sum _{t=0}^T{L}^{t+1}E\left[\left|{\tilde{Q}}_t-Q_{\tilde{\pi },t}\right||O_0=o_0\right]\end{array}$$

(Q_t, $Q_{\tilde{\pi },t}$ is used as abbreviation for Q_t (O_t, A_t), respectively $Q_{\tilde{\pi },t}$(O_t, A_t)). This completes the proof of the first result.

Start from (8) and use Hölder’s inequality to obtain, $\left|V_{\tilde{\pi }}(o_0)-V_{\pi }(o_0)\right|$

$$\begin{array}{l}\le 2\sum _{t=0}^T{E}_{\pi }\left[\underset{a_t}{max}\left|Q_t(O_t,A_{t-1},a_t)-{\tilde{Q}}_t(O_t,A_{t-1},a_t)\right||O_0=o_0\right]\\ +E_{\pi }\left[\underset{a_t}{max}\left|{\tilde{Q}}_t(O_t,A_{t-1},a_t)-Q_{\tilde{\pi },t}(O_t,A_{t-1},a_t)\right||O_0=o_0\right]\\ \le 2\sum _{t=0}^T\sqrt{E_{\pi }\left[\underset{a_t}{max}{\left|Q_t(O_t,A_{t-1},a_t)-{\tilde{Q}}_t(O_t,A_{t-1},a_t)\right|}^2|O_0=o_0\right]}\\ +\sqrt{E_{\pi }\left[\underset{a_t}{max}{\left|{\tilde{Q}}_t(O_t,A_{t-1},a_t)-Q_{\tilde{\pi },t}(O_t,A_{t-1},a_t)\right|}^2|O_0=o_0\right]}\\ \le 2\sum _{t=0}^T\sqrt{L{E}_{\pi }\left[\sum _{a_t}{\left|Q_t(O_t,A_{t-1},a_t)-{\tilde{Q}}_t(O_t,A_{t-1},a_t)\right|}^2{P}_t(a_t|O_t,A_{t-1})|O_0=o_0\right]}\\ +2\sum _{t=0}^T\sqrt{L{E}_{\pi }\left[\sum _{a_t}{\left|{\tilde{Q}}_t(O_t,A_{t-1},a_t)-Q_{\tilde{\pi },}{}_t(O_t,A_{t-1},a_t)\right|}^2{P}_t(a_t|O_t,A_{t-1})|O_0=o_0\right]}.\end{array}$$

Now use Lemma A1 and the lower bound on the p_t’s to obtain the result,

$$\begin{array}{l}\left|V_{\tilde{\pi }}(o_0)-V_{\pi }(o_0)\right|\le 2\sum _{t=0}^T\sqrt{LE\left[\prod _{\ell =0}^{t-1}\frac{1_{A_{\ell }={\pi }_{\ell }}}{p\ell (A\ell |O_{\ell },A_{\ell -1})}{(Q_t-{\tilde{Q}}_t)}^2|O_0=o_0\right]}\\ +\sqrt{LE\left[\prod _{\ell =0}^{t-1}\frac{1_{A_{\ell }={\pi }_{\ell }}}{p\ell (A\ell |O_{\ell },A_{\ell -1})}\hspace{0.17em}{({\tilde{Q}}_t-Q_{\tilde{\pi },t})}^2|O_0=o_0\right]}\\ \le 2L^{(t+1)/2}\sum _{t=0}^T\sqrt{E\left[{(Q_t-{\tilde{Q}}_t)}^2|O_0=o_0\right]}\\ +\sqrt{E\left[{({\tilde{Q}}_t-Q_{\tilde{\pi },t})}^2|O_0=o_0\right]}.\end{array}$$

▪

5. Finite Sample Upper Bounds on the Average Generalization Error

Traditionally the performance of a policy π is evaluated in terms of maximum generalization error: max_o [V^*(o) − V_π(o)] (Bertsekas and Tsitsiklis, 1996). However here we consider an average generalization error as in Kakade (2003) (see also Fiechter, 1997; Kearns, Mansour and Ng, 2000; Peshkin and Shelton, 2002); that is ∫_o[V^*(o) − V_π(o)] dF(o) for a specified distribution F on the observation space. The choice of F with density f = f₀ (f₀ is the density of O₀ in likelihoods (1) and (2)) is particularly appealing in the development of a policy in many medical and social science applications. In these cases, f₀ represents the distribution of initial observations corresponding to a particular population of subjects. The goal is to produce a good policy for this population of subjects. In general as in Kakade (2003) F may be chosen to incorporate domain knowledge concerning the steady state dis! tribution of a good policy. If only a training set of trajectories is available for learning and we are unwilling to assume that the system dynamics are Markovian, then the choice of F is constrained by the following consideration. If the distribution of O₀ in the training set (f₀) assigns mass zero to an observation o′, then the training data will not be able to tell us anything about V_π (o′). Similarly if f₀ assigns a very small positive mass to o′ then only an exceptionally large training set will permit an accurate estimate of V_π(o′). Of course this will not be a problem for the average generalization error, as long as F also assigns very low mass to o′. Consequently in our construction of the finite sample error bounds for the average generalization error, we will only consider distributions F for which the density of F, say f, satisfies ${sup}_o\left|\frac{f(o)}{f_0(o)}\right|\le M$for some finite constant M. In this case the average generalization error is bounded above by

$$\begin{array}{l}\int V^{*}(o)-V_{\pi }(o)\hspace{0.17em}dF(o)\hspace{0.17em}\le \hspace{0.17em}ME\hspace{0.17em}\left[V^{*}(O_0)-V_{\pi }(O_0)\right]\\ =-M{E}_{\pi }\left[\sum _{t=0}^T{\mu }_t^{*}(O_t,A_t)\right].\end{array}$$

The second line is a consequence of (5) and the fact that the distribution of O₀ is the same under likelihoods (1) and (2).

In the following theorem a non-asymptotic upper bound on the average generalization error is provided; this upper bound depends on the number of trajectories in the training set (n), the performance of the approximation on the training set, the complexity of the approximation space and of course on the confidence (δ) and accuracy (ɛ) demanded. The batch Q-learning algorithm minimizes quadratic forms (see (3)); thus we represent the performance of functions {Q₀, Q₁,..., Q_T} on the training set by these quadratic forms,

$${Err}_{n,Q_{t+1}}(Q_t)=E_n{\left[R_t+\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-Q_t(O_t,A_t)\right]}^2$$

for each j (recall Q_{T +1} is set to zero and E_n represents the expectation with respect to the probability obtained by choosing a trajectory uniformly from the training set).

The complexity of each Q_t space can be represented by it’s covering number (Anthony and Bartlett, 1999, pg 148). Suppose F is a class of functions from a space, X, to ℝ. For a sequence x = (x₁,..., x_n) ∈ Xⁿ, define F_|x to be a subset of ℝⁿ given by F_|x = {(f(x₁),..., f (x_n)): f ∈ F}. Define the metric dp on ℝⁿ by $d_p(z,y)={\left(1/n{{\sum }_{i=1}^n\left|z_i-y_i\right|}^p\right)}^{1/p}$ for p a positive integer (for p=∞, define $d_{\infty }(z,y)={max}_{i=1}^n\left|z_i-y_i\right|$). Then N (ɛ,F_|x, d_p) is defined as the minimum cardinality of an ɛ-covering of F_|x with respect to the metric d_p. Next given ɛ>0, positive integer n, metric d_p and function class, F, the covering number for F is defined as

$$N_p(\varepsilon ,F,n)=max\hspace{0.17em}\left\{N(\varepsilon ,F_{|x},d_p):x\in X^n\right\}.$$

In the following theorem, F = {max a_t+1 Q_t+1 (o_t+1, a_t ) - Q_t (o_t , a_t ) : Q_t ∈ Q_t , t = 0,...,T } and (x)⁺ is x if x > 0 and zero otherwise.

Theorem 1

Assume that the functions in Q_t , t ∈ 0,...,T are uniformly bounded. Suppose that there exists a positive constant, say L, for which p_t (a_t |o_t, a_t) ≥ L⁻¹ for all (o_t, a_t ) pairs, 0 ≥ t ≥ T . Then for ɛ > 0 and with probability at least 1–δ, over the random choice of the training set, every choice of functions, Q _j ∈ Q_j,j = 0,...,T with associated policy π defined by π_j(o_j, a _j–1) = arg max_aj Q_j(o_j,a_j–1) and every choice of functions, ${\tilde{Q}}_j\in \hspace{0.17em}{\text{Q}}_j,\hspace{0.17em}\hspace{0.17em}j=0,\dots ,T$ with associated policy $\tilde{\pi }$ defined by ${\tilde{\pi }}_j(o_j,a_{j-1})=\hspace{0.17em}arg\hspace{0.17em}{max}_{a_j}{\tilde{Q}}_j(o_j,a_j)$ the following bound is satisfied,

$$\begin{array}{l}\int \left|V_{\tilde{\pi }}(o)-V_{\pi }(o)\right|dF(o)\\ \le 6M{L}^{1/2}\sum _{t=0}^T{\left[\sum _{i=t}^T{(16)}^{i-t}{L}^i{(Er{r}_{n,Q_{i+1}}(Q_i)-Er{r}_{n,Q_{i+1}}({\tilde{Q}}_i))}^{+}\right]}^{1/2}\\ +12M{L}^{1/2}\varepsilon \\ {+\text{6ML}}^{\text{1}/\text{2}}\sum _{t=0}^T\sum _{i=t}^T{(16)}^{(i-t)/2}{L}^{i/2}\sqrt{E{\left[{\tilde{Q}}_i(O_i,A_i)-Q_{\tilde{\pi },i}(O_i,A_i)\right]}^2}.\end{array}$$

for n satisfying

$$4(T+1)N_1\left(\frac{{\varepsilon }^{\text{2}}}{32M^{\prime }{(16L)}^{(T+2)}},F,2n\right)\hspace{0.17em}exp\hspace{0.17em}\left\{\frac{{\varepsilon }^{\text{4}}n}{32{(M^{\prime })}^2{(16L)}^{2(T+2)}}\right\}\le \delta$$ (9)

and where M′ is a uniform upper bound on the absolute value of f ∈ F and E represents the expectation with respect to the distribution (1) generating the training set.

Remarks:

1. Suppose that Q^*_t ∈ Q_t for each select t. Select ${\tilde{Q}}_t=Q_t^{*}$ and ${\widehat{Q}}_t=arg\hspace{0.17em}{min}_{Q_t\in \hspace{0.17em}{Q}_t}\hspace{0.17em}Er{r}_{n,{\widehat{Q}}_{t+1}}(Q_t)$, t = T, T−1,..., 0 (recall Q_T+1, ${\widehat{Q}}_{T+1}$ are identically zero). Then with probability greater than 1 − δ, we obtain,

$$\int V^{*}(o)-V_{\widehat{\pi }}(o)dF(o)\le 12M{L}^{1/2}\varepsilon$$ (10)

for all n satisfying (9). Thus, as long as the covering numbers for each Q_t and thus for F do not grow too fast, estimating each Q_t by minimizing $Er{r}_{n,{\widehat{Q}}_{t+1}}(Q_t)$ yields a policy that consistently achieves the optimal value. Suppose the approximation spaces Q_t , t = 0,...,T are feed-forward neural networks as in remark 4 below. In this case the training set size n sufficient for (10) to hold need only be polynomial in (1/δ, 1/ɛ) and batch Q-learning is a probably approximate correct (PAC) reinforcement learning algorithm as defined by Fiechter (1997). As shown by Fiechter (1997) this algorithm can be converted to an efficient on-line reinforcement learning algorithm (here the word on-line implies updating the policy between trajectories).

2. Even when Q_t* does not belong to Q_t we can add the optimal Q function at each time, t, to the approximation space, Q_t with a cost of no more than an increase of 1 to the covering number $N_1\left(\frac{{\varepsilon }^{\text{2}}}{32M^{\prime }{(16L)}^{(T+2)}},F,2n\right)$. If we do this the result continues to hold when we set $\tilde{\pi }$ to an optimal policy ${\pi }^{*}$ and set ${\tilde{Q}}_t=Q_t^{*}$ for each t; the generalization error is

$$\begin{array}{l}\int V^{*}(o)-V_{\pi }(o)dF(o)\le 6M{L}^{1/2}\sum _{t=0}^T{\left[\sum _{i=t}^T{(16)}^{i-t}{L}^{i}Er{r}_{n,Q_{i+1}}(Q_i)\right]}^{1/2}\\ +12M{L}^{1/2}\varepsilon \end{array}$$

for all n satisfying (9). This upper bound is consistent with the practice of using a policy $\widehat{\pi }$ for which ${\widehat{\pi }}_t(o_t,a_{t-1})\in \hspace{0.17em}arg\hspace{0.17em}{max}_{a_t}{\widehat{Q}}_t(o_t,a_t)$ and ${\widehat{Q}}_t\in \hspace{0.17em}arg\hspace{0.17em}{min}_{Q_t\in Q_t}{Err}_{n,{\widehat{Q}}_{t+1}}(Q_t).$ Given that the covering numbers for the approximation space can be expressed in a sufficiently simple form (as in remark 4 below), this upper bound can be used to carry out model selection using structural risk minimization (Vapnik, 1982). That is, one might consider a variety of approximation spaces and use structural risk minimization to use the training data to choose which approximation space is best. The resulting upper bound on the average generalization error can be found by using the above result and Lemma 15.5 of Anthony and Bartlett (1999).

3. The restriction on n in (9) is due to the complexity associated with the approximation space (e.g. the Q_t ’s). The restriction is crude; to see this, note that if there were only a finite number of functions in F then n need only satisfy

$$2(T+1)\left|F\right|exp\hspace{0.17em}\left\{\frac{2{\varepsilon }^{\text{4}}n}{{(3M^{\prime })}^2{(16L)}^{2(T+2)}}\right\}=\delta$$

(use Hoeffding’s inequality; see Anthony and Bartlett, pg 361, 1999) and thus for a given (ε, &δ) we may set the number of trajectories in the training set n equal to $\frac{{(3M^{\prime })}^2{(16L)}^{2(T+2)}}{2{\varepsilon }^{\text{4}}}\hspace{0.17em}ln\hspace{0.17em}\left(\frac{2(T+1)\left|F\right|}{\delta }\right).$This complexity term appears similar to that achieved by learning algorithms (e.g. see Anthony and Bartlett, 1999, pg. 21) or in reinforcement learning (e.g. Peshkin and Shelton, 2002) however note that n is of the order ɛ⁻⁴ rather than the usual ɛ⁻². The ɛ⁻⁴ term (instead of ɛ⁻²) is attributable to the fact that Err_Qt+1(Q_t) is not only a function of Q_t but also of Q_t+1. However further assumptions on the approximation space permit an improved result. See Theorem 2 below for one possible refinement. Note the needed training set size n depends exponentially on the horizon time T but not on the dimension of the observation space. Thi! s is not unexpected as the upper bounds on the generalization error of both Kearns, Mansour and Ng (2000) and Peshkin and Shelton’s (2002) policy search methods (the latter using a training set and importance sampling weights) also depend exponentially on the horizon time.

4. When F is infinite, we use covering numbers for the approximation space Q_t and then appeal to Lemma A2 in the appendix to derive a covering number for F; this results in

$$N_1(\varepsilon ,F,n)\le (T+1)\hspace{0.17em}\underset{t=0,\dots ,T}{max}{N}_1{\left(\frac{\varepsilon }{\text{2}\left|A\right|},Q_t,\left|A\right|n\right)}^2.$$

One possible approximation space is based on feed-forward neural networks. From Anthony and Bartlett (1999) we have that if each Q_j is the class of functions computed by a feed-forward network with W weights and k computation units arranged in L layers and each computation unit has a fixed piecewise-polynomial activation function with q pieces and degree no more than ℓ, then $N_1(\varepsilon ,Q_t,n)\le e(d+1){\left(\frac{2e{M}^{\prime }}{\varepsilon }\right)}^d$ where d = 2(W + 1)(L + 1) log₂(4(W + 1)(L + 1)q(k + 1)/ln 2) + 2(W + 1)(L + 1)² log₂(ℓ + 1) + 2(L + 1). To see this combine Anthony and Bartlett’s Theorems 8.8, 14.1 and 18.4. They provide covering numbers for functions computed by other types of neural networks as well. A particularly simple neural network is an affine combination of a given set of p input features; i.e. $f(x)={\omega }_{\text{0}}+{\sum }_{i=1}^{p-1}{\omega }_i{x}_i$ for (1, x) a vector of p real valued features and each ω_i ∈ ℝ. Suppose each Q_t is a class of functions computed by this network. Then Theorems 11.6 and 18.4 of Anthony and Bartlett imply that $N_1(\varepsilon ,Q_t,n)\le e(p+1){\left(\frac{2e{M}^{\prime }}{\varepsilon }\right)}^p$. In this case

$$n\ge \frac{32{(M^{\prime })}^4{(16L)}^{2(T+2)}}{{\varepsilon }^{\text{4}}}\hspace{0.17em}log\hspace{0.17em}\left(\frac{4{(T+1)}^2\hspace{0.17em}{e}^2{(p+1)}^2{(128e\left|A\right|{(M^{\prime })}^2{(16L)}^{(T+2)})}^{2p}}{{\delta \varepsilon }^{\text{4}p}}\right).$$

This number will be large for any reasonable accuracy, ɛ and confidence, δ.

Proof of Theorem 1

An upper bound on the average difference in value functions can be obtained from Lemma 2 by using Jensen’s inequality and the assumption that the the density of F ( f ) satisfies ${sup}_o\left|\frac{f(o)}{f_0(o)}\right|\le M$ for some finite constant M:

$$\begin{array}{l}\int \left|V_{\tilde{\pi }}(o)-V_{\pi }(o)\right|\hspace{0.17em}dF(o)\hspace{0.17em}\le \hspace{0.17em}M\sum _{t=0}^{T}2L^{(t+1)/2}\sqrt{E{\left[Q_t(O_i,A_i)-{\tilde{Q}}_t(O_i,A_i)\right]}^2}\\ +M\sum _{t=0}^{T}2L^{(t+1)/2}\sqrt{E{\left[{\tilde{Q}}_t-Q_{\tilde{\pi },t}\right]}^2}\end{array}$$ (11)

where ${\tilde{Q}}_t-Q_{\tilde{\pi },t}$ is used as abbreviation for ${\tilde{Q}}_t(O_t,A_t)$, respectively $Q_{\tilde{\pi },t}(O_t,A_t)$. In the following an upper bound on each $E{\left[Q_t-{\tilde{Q}}_t\right]}^2$ is constructed.

The performance of the approximation on an infinite training set can be represented by

$$Er{r}_{Q_{t+1}}(Q_t)=E{\left[R_t+\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-Q_t\right]}^2$$

for each t (recall Q_T+1 = 0, also we abbreviate Q_t (O_t, A_t ) by Q_t whenever no confusion may arise). The errors, Err’s, can be used to provide an upper bound on the L₂ norms on the Q-functions by the following argument. Consider $Er{r}_{Q_{t+1}}(Q_t)-Er{r}_{Q_{t+1}}({\tilde{Q}}_t)$ for each t. Within each of these quadratic forms add and subtract

$$Q_{\tilde{\pi },t+1}(O_{t+1},A_t,{\tilde{\pi }}_{t+1})-Q_{\tilde{\pi },t}-E\left[\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-Q_{\tilde{\pi },t+1}(O_{t+1},A_t,{\tilde{\pi }}_{t+1})|O_t,A_t\right].$$

In the above $Q_{\tilde{\pi },t+1}(O_{t+1},A_t,{\tilde{\pi }}_{t+1})$ is defined as $Q_{\tilde{\pi },t+1}(O_{t+1},A_t,a_{t+1})$ with a_t+1 replaced by ${\tilde{\pi }}_{t+1}(O_{t+1},A_t).$ Expand each quadratic form and use the fact that $E\left[R_t+Q_{\tilde{\pi },t+1}(O_{t+1},A_t,{\tilde{\pi }}_{t+1})|O_t,A_t\right]=Q_{\tilde{\pi },t}$. Cancelling common terms yields

$$\begin{array}{l}E{\left[Q_{\tilde{\pi },t}-Q_t+E\left[\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-Q_{\tilde{\pi },t+1}(O_{t+1},A_t,{\tilde{\pi }}_{t+1})|O_t,A_t\right]\right]}^2\\ -E{\left[Q_{\tilde{\pi },t}-{\tilde{Q}}_t+E\left[\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-Q_{\tilde{\pi },t+1}(O_{t+1},A_t,{\tilde{\pi }}_{t+1})|O_t,A_t\right]\right]}^2.\end{array}$$

Add and subtract ${\tilde{Q}}_t$ in the first quadratic form and expand. This yields

$$\begin{array}{l}Er{r}_{Q_{t+1}}(Q_t)-Er{r}_{Q_{t+1}}({\tilde{Q}}_t)=\\ E{[{\tilde{Q}}_t-Q_t]}^2+2E\hspace{0.17em}[{\tilde{Q}}_t-Q_t]\hspace{0.17em}[{\tilde{Q}}_t-Q_{\tilde{\pi },t}]\\ +2E\left[\left({\tilde{Q}}_t-Q_t\right)\left(\underset{a_{t+1}}{max}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-\underset{a_{t+1}}{max}{\tilde{Q}}_{t+1}(O_{t+1},A_t,a_{t+1})\right)\right]\\ +2E\left[\left({\tilde{Q}}_t-Q_t\right)\left(\underset{a_{t+1}}{max}{\tilde{Q}}_{t+1}(O_{t+1},A_t,a_{t+1})-Q_{\tilde{\pi },t+1}(O_{t+1},A_t,{\tilde{\pi }}_{t+1})\right)\right].\end{array}$$ (12)

Using the arguments similar to those used around Equation (8) and using the fact that (x + y)² ≤ 2x² + 2y² we obtain,

$$\begin{array}{l}Er{r}_{Q_{t+1}}(Q_t)-Er{r}_{Q_{t+1}}({\tilde{Q}}_t)\ge E{\left[Q_t-{\tilde{Q}}_t\right]}^2\\ -4{\left(E{\left[Q_t-{\tilde{Q}}_t\right]}^2\left(E{\left[{\tilde{Q}}_t-Q_{\tilde{\pi },t}\right]}^2+LE{\left[Q_{t+1}-{\tilde{Q}}_{t+1}\right]}^2+LE{\left[{\tilde{Q}}_{t+1}-Q_{\tilde{\pi },t+1}\right]}^2\right)\right)}^{1/2}.\end{array}$$

Using this inequality we can now derive an upper bound on each $E{\left[Q_t-{\tilde{Q}}_t\right]}^2$ in terms of the Err’s and the $E\left[{\tilde{Q}}_{t+1}-Q_{\tilde{\pi },t+1}\right]$’s. Define

$$m_t=L^{-(T-t)}E{\left[{\tilde{Q}}_t-Q_{\tilde{\pi },t}\right]}^2\mathrm{\hspace{0.17em} }\text{and\hspace{0.17em}}{b}_t=L^{-(T-t)}E{\left[Q_t-{\tilde{Q}}_t\right]}^2$$

and

$$e_t=L^{-(T-t)}\left(Er{r}_{Q_{t+1}}(Q_t)-Er{r}_{Q_{t+1}}({\tilde{Q}}_t)\right)$$

for t ≤ T and b_{T +1} = m_{T +1} = e_{T +1} = 0. We obtain

$$e_t\ge b_t-4\sqrt{b_t(m_t+b_{t+1}+m_{t+1}).}$$

Completing the square, reordering terms, squaring once again and using the inequality (x + y)² ≤ 2x² + 2y² yields b_t ≤ 16(b_t+1 + m_t + m_t+1) + 2e_t for t ≤ T. We obtain

$$b_{T-t}\le 2\sum _{i=0}^t{(16)}^i{e}_{T-t+i}+\sum _{i=1}^t{(16)}^i(16+1)m_{T-t+i}+16m_{T-t}.$$

Inserting the definitions of b_T−t, e_T−t+i and reordering, yields

$$\begin{array}{l}E{\left[Q_t-{\tilde{Q}}_t\right]}^2\hspace{0.17em}\le 2\sum _{i=t}^T{(16L)}^{i-t}\left(Er{r}_{Q_{i+1}}(Q_i)-Er{r}_{Q_{i+1}}({\tilde{Q}}_i)\right)\\ +\sum _{i=t+1}^T{(16)}^{i-t}(16+1)L^{T-t}{m}_i+L^{T-t}{m}_t.\end{array}$$ (13)

As an aside we can start from (12) and derive the upper bound,

$$\begin{array}{l}Er{r}_{Q_{t+1}}(Q_t)-Er{r}_{Q_{t+1}}({\tilde{Q}}_t)\le E{\left[Q_t-{\tilde{Q}}_t\right]}^2\\ +4L^{(T-t)}\sqrt{L^{-(T-t)}E{\left[Q_t-{\tilde{Q}}_t\right]}^2\left(m_t+L^{-(T-t-1)}E{\left[Q_{t+1}-{\tilde{Q}}_{t+1}\right]}^2+m_{t+1}\right).}\end{array}$$

This combined with (13) implies that minimizing each $Er{r}_{Q_{t+1}}(Q_t)-Er{r}_{Q_{t+1}}({\tilde{Q}}_t)$ in Q_t is equivalent to minimizing each $E{\left[Q_t-{\tilde{Q}}_t\right]}^2$ in Q_t modulo the approximation terms m_t for t = 0,..., T.

Returning to the proof next note that

$$\begin{array}{l}Er{r}_{Q_{t+1}}(Q_t)-Er{r}_{Q_{t+1}}({\tilde{Q}}_t)\le \left|Er{r}_{Q_{t+1}}(Q_t)-Er{r}_{n,Q_{t+1}}(Q_t)\right|\\ +\left|Er{r}_{Q_{t+1}}({\tilde{Q}}_t)-Er{r}_{n,Q_{t+1}}({\tilde{Q}}_t)\right|\\ +{\left(Er{r}_{n,Q_{t+1}}(Q_t)-Er{r}_{n,Q_{t+1}}({\tilde{Q}}_t)\right)}^{+}\end{array}$$

where (x)⁺ is equal to x if x ≥ 0 and is equal to 0 otherwise. Note that if each Q_t minimizes Err_n,Qt+1 as in (3) then the third term is zero. Substituting into (13), we obtain

$$\begin{array}{l}E{\left[Q_t-{\tilde{Q}}_t\right]}^2\le 2\sum _{i=t}^T{(16L)}^{i-t}(\left|Er{r}_{Q_{i+1}}(Q_i)-Er{r}_{n,Q_{i+1}}({\tilde{Q}}_i)\right|\\ +\left|Er{r}_{Q_{i+1}}({\tilde{Q}}_i)-Er{r}_{n,Q_{i+1}}({\tilde{Q}}_i)\right|\\ +{\left(Er{r}_{n,Q_{i+1}}(Q_i)-Er{r}_{n,Q_{i+1}}({\tilde{Q}}_i)\right)}^{+})\\ +\sum _{i=t+1}^T{(16)}^{i-t}(16+1)L^{i-t}E{\left[{\tilde{Q}}_i-Q_{\tilde{\pi },i}\right]}^2+E{\left[{\tilde{Q}}_t-Q_{\tilde{\pi },t}\right]}^2.\end{array}$$

Combine this inequality with (11); simplify the sums and use the fact that for x, y both nonnegative $\sqrt{x+y}\le \sqrt{x}+\sqrt{y}$ to obtain $\int \left|V_{\tilde{\pi }}(o)-V_{\pi }(o)\right|dF(o)$

$$\begin{array}{l}\le 6M{L}^{1/2}{\sum _{t=0}^T\left[\sum _{i=t}^T{(16)}^{i-t}{L}^i{\left(Er{r}_{n,Q_{i+1}}(Q_i)-Er{r}_{n,Q_{i+1}}({\tilde{Q}}_i)\right)}^{+}\right]}^{1/2}\\ +12M{L}^{1/2}{(16L)}^{(T+2)/2}\sqrt{\underset{t}{max}\underset{Q_t,Q_{t+1}}{\hspace{0.17em}sup}\left|Er{r}_{Q_{t+1}}(Q_t)-Er{r}_{n,Q_{t+1}}(Q_t)\right|}\\ +6M{L}^{1/2}\sum _{t=0}^T\sum _{i=t}^T{(16)}^{(i-t)/2}{L}^{i/2}\sqrt{E{\left[{\tilde{Q}}_i-Q_{\tilde{\pi },i}\right]}^2}.\end{array}$$

All that remains is to provide an upper bound on

$$P\left[\underset{i=0}{\overset{T}{\cup }}\left\{\text{for\hspace{0.17em}}{\text{some}}_{Q_t\in Q_t,t=0,\dots ,T}\left|Er{r}_{Q_{t+1}}(Q_i)-Er{r}_{n,Q_{i+1}}(Q_i))\right|>{\varepsilon }^{\prime }\right\}\right].$$

This probability is in turn bounded above by

$$\sum _{i=0}^{T}P\left[\text{for\hspace{0.17em}}{\text{some}}_{Q_t\in Q_t,t=0,\dots ,T}\left|Er{r}_{Q_{i+1}}(Q_i)-Er{r}_{n,Q_{i+1}}(Q_i)\right|>{\varepsilon }^{\prime }\right].$$

Anthony and Bartlett (1999, pg. 241) use Hoeffding’s inequality along with the classical techniques of symmetrization and permutation to provide the upper bound (see also van der Vaart and Wellner, 1996),

$$\begin{array}{l}P\left[\text{for\hspace{0.17em}}{\text{some}}_{Q_t\in Q_t,t=0,\dots ,T}\left|Er{r}_{Q_{i+1}}(Q_i)-Er{r}_{n,Q_{i+1}}(Q_i)\right|>{\varepsilon }^{\prime }\right]\\ \le 4N_1\left(\frac{{\varepsilon }^{\prime }}{32M^{\prime }},F,2n\right)\hspace{0.17em}exp\hspace{0.17em}\left\{-\frac{{\text{(}{\varepsilon }^{\prime }\text{)}}^{\text{2}}n}{32{(M^{\prime })}^2}\right\}.\end{array}$$

Put $\varepsilon ={(16L)}^{(T+2)/2}\sqrt{{\varepsilon }^{\prime }}$ to obtain the results of the theorem.

Suppose the Q functions are approximated by linear combinations of p features; for each t = 0,...,T, denote the feature vector by q_t (o_t, a_t ). The approximation space is then,

$$Q_t=\{Q_t(o_t,a_t)={\theta }^T{q}_t(o_t,a_t)\hspace{0.17em}:\hspace{0.17em}\theta \in \hspace{0.17em}\mathrm{\Theta }\}$$

where Θ is a subset of R^p. In this case, the batch Q-learning algorithm may be based on (4); we represent the performance of the functions {Q₀,..., Q_t } on the training set by

$${\widetilde{Err}}_{n,Q_{t+1}}(Q_t)=E_n\left[\left(R_t+\underset{a_{t+1}}{max}\hspace{0.17em}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-Q_t(O_t,A_t)\right)q_t(O_t,A_t)\right]$$

for t = 0,...,T (recall E_n represents the expectation with respect to the probability obtained by choosing a trajectory uniformly from the training set). In this theorem

$$F^{\prime }=\underset{i=1}{\overset{p}{\cup }}\underset{t=1}{\overset{T}{\cup }}\left\{\left(r_t+\underset{a_{t+1}}{max}\hspace{0.17em}{Q}_{t+1}(o_{t+1},a_{t+1};{\theta }_{t+1})-Q_t(o_t,a_t;{\theta }_t)\right)q_{ti}(o_t,a_t\text{)\hspace{0.17em}}:{\theta }_t{,\theta }_{t+1}\hspace{0.17em}\in \hspace{0.17em}\mathrm{\Theta }\right\}$$.

Define the functions $\{{\overline{Q}}_0,\dots ,{\overline{Q}}_T\}$, and the policy, $\overline{\pi }$, as follows. First define ${\overline{Q}}_T(O_T,A_T)$ to be the projection of E [R_T |O_T, A_T ] on the space spanned by q_T. Then set ${\overline{\pi }}_T(o_T,a_{T-1})\hspace{0.17em}\in \hspace{0.17em}\hspace{0.17em}arg\hspace{0.17em}{max}_{a_T}{\overline{Q}}_T(o_T,a_T)$. Next for t = T − 1,...,0, set ${\overline{Q}}_t(O_t,A_t)$as the projection of $E\left[R_t+{\overline{Q}}_{t+1}(O_{t+1},A_t,{\overline{\pi }}_{t+1})|O_t,A_t\right]$on the space spanned by q_t (recall ${\overline{Q}}_{t+1}(O_{t+1},A_t,{\overline{\pi }}_{t+1})$ is defined as ${\overline{Q}}_{t+1}(O_{t+1},A_t,a_{t+1})$ with a_t+1 replaced by ${\overline{\pi }}_{t+1}(O_{t+1},A_t))$. And set ${\overline{\pi }}_t(o_t,a_{t-1})\hspace{0.17em}\in \hspace{0.17em}arg{max}_{a_t}{\overline{Q}}_t(o_t,a_t)$. These projections are with respect to P, the distribution which generated the trajectories in the training set (the likelihood is in (1)).

Theorem 2

Suppose that there exists a positive constant, say L, for which p_t (a_t |o_t, a_t−1) ≥ L⁻¹ for all (o_t, a_t−1), 0 ≤ t ≤ T. Suppose that for each t, x ∈ R^p, x^T Eq_t q_t^T x > η ||x||² where η > 0 (|| · || is the Euclidean norm). Also assume that Θ is a closed subset of {x ∈ R^p : ||x|| ≤ M_Θ} and for all (t, i), the ith component in the vector q_t is pointwise bounded; |q_ti| ≤M_Q for M_Q a constant. Then for ɛ > 0, with probability at least 1 – δ, over the random choice of the training set, every choice of functions, Q_t ∈ Q_t and functions ${\tilde{Q}}_t,t=0,\dots ,T$ with associated policies defined by π with π_t (o_t , a_t–1 ∈ arg max_at Q_t(o_t , a_t) and $\tilde{\pi }$ with ${\overline{\pi }}_t(o_t,a_{t-1})\hspace{0.17em}\in \hspace{0.17em}arg{max}_{a_t}{\tilde{Q}}_t(o_t,a_t)$ respectively, the following bounds are satisfied

$$\begin{array}{l}\sum _{t=0}^T{L}^{(t+1)}E\left|{\overline{Q}}_t(O_t,A_t)-Q_t(O_t,A_t)\right|\hspace{0.17em}\le \hspace{0.17em}\sqrt{p}{M}_Q/\eta \sum _{t=0}^T{L}^{(t+1)}\sum _{j=t}^T{(Lp{M}_Q^2/\eta )}^{j-t}\hspace{0.17em}\Vert {\widetilde{Err}}_{n,Q_{j+1}}(Q_j)\Vert \\ +4\varepsilon \text{.}\end{array}$$

for t = 0,...,T , where E represents the expectation with respect to the distribution (1) generating the training set and

$$\begin{array}{l}\int \left|V_{\tilde{\pi }}(o)-V_{\pi }(o)\right|dF(o)\hspace{0.17em}\le \hspace{0.17em}2M\sqrt{p}{M}_Q/\eta \sum _{t=0}^T{L}^{(t+1)}\sum _{j=t}^T{(Lp{M}_Q^2/\eta )}^{j-t}\hspace{0.17em}\Vert {\widetilde{Err}}_{n,Q_{j+1}}(Q_j)\Vert \\ +8M\varepsilon \\ +2M\sum _{t=0}^T{L}^{(t+1)}E\left|{\overline{Q}}_t(O_t,A_t)-{\tilde{Q}}_t(O_t,A_t)\right|\\ +2M\sum _{t=0}^T{L}^{(t+1)}E\left|{\tilde{Q}}_t(O_t,A_t)-Q_{\tilde{\pi },t}(O_t,A_t)\right|\end{array}$$

for n larger than

$${\left(\frac{C}{\varepsilon }\right)}^2\hspace{0.17em}log\hspace{0.17em}\left(\frac{B}{\delta }\right)$$ (14)

where $C=4\sqrt{2}{M}^{\prime }{p}^{T+1/2}{M}_Q^{2T+1}{\eta }^{-(T+1)}{L}^{T+1},\hspace{0.17em}{M}^{\prime }$ is a uniform upper bound on the absolute value on all f ∈ F′ and $B={\varepsilon }^{-2p}{4}^{6p+3}{p}^{2T\hspace{0.17em}p+p+3}{(T+1)}^2{e}^{2p+2}{(M^{\prime })}^{4p}{\left|A\right|}^p{M}_Q^{{}_{{}^{(2T+1)2p}}}{\eta }^{-2p(T+1)}{L}^{2p(T+1)}$

Remarks:

1.Define ${\widehat{Q}}_t$ as a zero of ${\widetilde{Err}}_{n,{\widehat{Q}}_{t+1}}(Q_t),t=T,T-1,\dots ,0$ (recall that ${\widehat{Q}}_{T+1}$ is identically zero). Suppose that Q_t* ∈Q_t for each t; in this case ${\overline{Q}}_t=Q_t^{*}$ for all t (we ignore sets of measure zero in this discussion). Then with probability greater than 1 – δ and $\tilde{\pi }={\pi }^{*},\hspace{0.17em}{\tilde{Q}}_t=Q_t^{*}$ we obtain

$$\int V^{*}(o)-V_{\widehat{\pi }}(o)dF(o)\hspace{0.17em}\le \hspace{0.17em}8M\varepsilon$$

for all n satisfying (14). Thus estimating each Q_t by solving ${\widetilde{Err}}_{n,Q_{t+1}}(Q_t)=0,\hspace{0.17em}\hspace{0.17em}t=T,\dots ,0,$ yields a policy that consistently achieves the optimal value.

2. Again define ${\widehat{Q}}_t$ as a zero of ${\widetilde{Err}}_{n,{\widehat{Q}}_{t+1}}(Q_t),\hspace{0.17em}\hspace{0.17em}t=T,T-1,\dots ,0.$Given two T + 1 vectors of functions Q′ = {Q′_0,…, Q′_T} and Q = {Q_0,…, Q_T} define $\ell (Q^{\prime },Q)={\sum }_{t=0}^T{L}^{t+1}E\left|{Q^{\prime }}_t(O_t,A_t)-Q_t(O_t,A_t)\right|.$ Then the first result of Theorem 2 implies that $\ell (\overline{Q},\widehat{Q})$ converges in probability to zero. From Lemma 2 we have that $\int \left|V_{\tilde{\pi }}(o)-V_{\pi }(o)\right|\hspace{0.17em}dF(o)\le 2M\ell (Q,\tilde{Q})+2M\ell (Q_{\tilde{\pi }},\tilde{Q})$ and thus $\int \left|V_{\tilde{\pi }}(o)-V_{\widehat{\pi }}(o)\right|\hspace{0.17em}dF(o)$is with high probability bounded above by $2M\ell (\overline{Q},\tilde{Q})+2M\ell (Q_{\tilde{\pi }},\tilde{Q}).$ Consequently the presence of the third and fourth terms in Theorem 2 is not surprising. It is unclear whether the “go-between” ${\tilde{Q}}_t$ is necessary.

3. Recall the space of policies implied by the approximation spaces for the Q-functions is given by ${\mathrm{\Pi }}_Q=\{{\pi }_{\theta },\theta \in \mathrm{\Theta }\}$ where π_θ = {π_1,θ,…,π_T,θ} and where each π_t,θ (o_t, a_t–1) ∈ arg max _at Q_t (o_t , a_t;θ) for some Q_t ∈ Q_t. Suppose that max _{π ∈ ΠQ} ∫V_π(o)dF(o) is achieved by some member of Π_Q and $\tilde{\pi }\hspace{0.17em}\in \hspace{0.17em}arg\hspace{0.17em}{max}_{\pi \in \hspace{0.17em}{\mathrm{\Pi }}_Q}\int V_{\pi }(o)dF(o).$ Ideally Q-learning would provide a policy that achieves the highest value as compared to other policies in Π_Q (as is the case with $\tilde{\pi }$). This is not necessarily the case. As discussed in the above remark batch Q-learning yields estimated Q-functions for which $\ell (\overline{Q},\widehat{Q})$ converges to zero. The policy $\overline{\pi }$ may not produce a maximal value; that is $\int V_{\tilde{\pi }}(o)-V_{\overline{\pi }}(o)dF(o)$need not be zero (see also the remark following Lemma 2). Recall from Lemma 2 that $2M\ell (\overline{Q},\tilde{Q})+2M\ell (\tilde{Q},Q_{\tilde{\pi }})$ is an upper bound on this difference. It is not hard to see that $\ell (\tilde{Q},Q_{\tilde{\pi }})$ is zero if and only if $\tilde{\pi }$ is the optimal policy; indeed the optimal Q-function would belong to the approximation space. The Q-learning algorithm does not directly maximize the value function. As remarked in Tsitsiklis and van Roy (1997) the goal of the Q-learning algorithm is to construct an approximation to the optimal Q-function within the constraints imposed by the app! roximation space; this approximation is a projection when the approximation space is linear. Approximating the Q-function yields an optimal policy if the approximating class is sufficiently rich. Ormoneit and Sen (2002) consider a sequence of approximation spaces (kernel based spaces indexed by a bandwidth) and make assumptions on the optimal value function which guarantee that this sequence of approximations spaces is sufficient rich (as the bandwidth decreases with increasing training set size) so as to approximate the optimal value function to any desired degree.

4. Again define ${\widehat{Q}}_t$ as a zero of ${\widetilde{Err}}_{n,{\widehat{Q}}_{t+1}}(Q_t),t=T,T-1,\dots ,0.$ Since $\ell (\overline{Q},\widehat{Q})$ converges in probability to zero, one might think that $\int \left|V_{\overline{\pi }}(o)-V_{\widehat{\pi }}(o)\right|dF(o)$ should be small as well. Referring to Lemma 1, we have that the difference in value functions $\int \left|V_{\overline{\pi }}(o)-V_{\widehat{\pi }}(o)\right|dF(o)$ can be expressed as the sum over t of the expectation of $Q_{\tilde{\pi },t}(O_t,A_{t-1},{\widehat{\pi }}_t)-Q_{\overline{\pi },t}(O_t,A_{t-1},{\overline{\pi }}_t).$However $\ell (\overline{Q},\widehat{Q})$ small does not imply that $\widehat{\pi }$ and $\overline{\pi }$ will be close nor does it imply that $Q_{\overline{\pi },t}(O_t,A_{t-1},{\widehat{\pi }}_t)-Q_{\overline{\pi },t}(O_t,A_{t-1},{\overline{\pi }}_t)$ will be small. To see the former consider an action space with 10 actions, 1,...,10 and ${\widehat{Q}}_t(o_t,a_t)=1$ for a = 1,...,9, ${\widehat{Q}}_t(o_t,10)=1+1/2\varepsilon$ and ${\overline{Q}}_t(o_t,a_t)=1-1/2\varepsilon \text{\hspace{0.17em}for\hspace{0.17em}}a=2,\dots ,10,\hspace{0.17em}{\overline{Q}}_t(o_t,1)=1.$So ${\overline{Q}}_t$ and ${\widehat{Q}}_t$ are uniformly less than ɛ apart yet the argument of their maxima are 1 and 10.

Proof of Theorem 2

Fix Q_t = θ_t^T q_t, θ∈ θ for t = 0,...,T . Define an infinite training sample version of ${\widetilde{Err}}_n$ as

$$\begin{array}{l}{\widetilde{Err}}_{Q_{t+1}}(Q_t)=E\left[\left(R_t+\underset{a_{t+1}}{max}\hspace{0.17em}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-Q_t\right)q_t\right]\\ =E\left[\left({\overline{Q}}_t+\underset{a_{t+1}}{max}\hspace{0.17em}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-{\overline{Q}}_{t+1}(O_{t+1},A_t,{\overline{\pi }}_{t+1})-Q_t\right)q_t\right]\end{array}$$

where Q_t is an abbreviation for Q_t (O_t, A_t). To derive the last equality recall that ${\overline{Q}}_t(O_t,A_t)$ is the projection of $E\left[R_t+{\overline{Q}}_{t+1}(O_{t+1},A_t,{\overline{\pi }}_{t+1})|O_t,A_t\right]$ on the space spanned by q_t . Since ${\overline{Q}}_t$ is a projection we can write $Q_t={\theta }_t^T{q}_t$ for some ${\theta }_{\overline{\pi },t}\in \mathrm{\Theta }.$ Also we can write Q_t = θ^T_t q_t for some θ_t ∈ θ. The $\widetilde{Err}$’s provide a pointwise upper bound on the differences, $\left|{\overline{Q}}_t-Q_t\right|$, as follows. Rearrange the terms in ${\widetilde{Err}}_{Q_{t+1}}$using the fact that Eq_t q_t^T is invertible to obtain

$$\begin{array}{l}{\text{(}\theta }_{\overline{\pi },t}-{\theta }_t)={(E{q}_t{q}_t^T)}^{-1}\hspace{0.17em}{\widetilde{Err}}_{Q_{t+1}}(Q_t)\\ -{(E{q}_t{q}_t^T)}^{-1}E\left[\left(\underset{a_{t+1}}{max}\hspace{0.17em}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-{\overline{Q}}_{t+1}(O_{t+1},A_t,{\overline{\pi }}_{t+1})\right)q_t\right].\end{array}$$

Denote the Euclidean norm of a p dimensional vector x by ||x||. Then

$$\begin{array}{l}\left|{({\theta }_{\overline{\pi },t}-{\theta }_t)}^T{q}_t\right|\hspace{0.17em}\le \hspace{0.17em}(1/\eta )\Vert {\widetilde{Err}}_{Q_{t+1}}(Q_t)\Vert \hspace{0.17em}\Vert q_t\Vert +\\ (1/\eta )E\left[\left|\underset{a_{t+1}}{max}\hspace{0.17em}{Q}_{t+1}(O_{t+1},A_t,a_{t+1})-{\overline{Q}}_{t+1}(O_{t+1},A_t,{\overline{\pi }}_{t+1})\right|\Vert q_t\Vert \right]\Vert q_t\Vert \\ \le (1/\eta )\Vert {\widetilde{Err}}_{Q_{t+1}}(Q_t)\Vert \hspace{0.17em}\Vert q_t\Vert +(1/\eta )LE\left[\left|Q_{t+1}-{\overline{Q}}_{t+1}\right|\Vert q_t\Vert \right]\Vert q_t\Vert \hspace{0.17em}\\ \le (1/\eta )\sqrt{p}{M}_Q\Vert {\widetilde{Err}}_{Q_{t+1}}(Q_t)\Vert \hspace{0.17em}+(1/\eta )Lp{M}_Q^{2}E\left[\left|Q_{t+1}-{\overline{Q}}_{t+1}\right|\right]\hspace{0.17em}\end{array}$$

for t ≤ T . To summarize

$$E\left|{\overline{Q}}_t-Q_t\right|\le (1/\eta )\sqrt{p}{M}_Q\Vert {\widetilde{Err}}_{Q_{t+1}}(Q_t)\Vert \hspace{0.17em}+(1/\eta )Lp{M}_Q^{2}E\left[\left|Q_{t+1}-{\overline{Q}}_{t+1}\right|\right]\hspace{0.17em}$$

where Q_t, ${\overline{Q}}_t$ is an abbreviation for Q_t (O_t, A_t), respectively ${\overline{Q}}_t(O_t,A_t)$, for each t.

As in the proof of Theorem 1, these inequalities can be solved for each $\text{E}\left|{\overline{Q}}_t-Q_t\right|$ to yield

$$\begin{array}{l}E\left|{\overline{Q}}_t-Q_t\right|\le \left(\sqrt{p}{M}_Q/\eta \right)\sum _{j=t}^T{\left(Lp{M}_Q^2/\eta \right)}^{j-t}\Vert {\widetilde{Err}}_{Q_{j+1}}(Q_j)\Vert \\ \le \left(\sqrt{p}{M}_Q/\eta \right)\sum _{j=t}^T{\left(Lp{M}_Q^2/\eta \right)}^{j-t}\Vert {\widetilde{Err}}_{n,Q_{j+1}}(Q_j)-{\widetilde{Err}}_{Q_{j+1}}(Q_j)\Vert \\ +\left(\sqrt{p}{M}_Q/\eta \right)\sum _{j=t}^T{\left(Lp{M}_Q^2/\eta \right)}^{j-t}\Vert {\widetilde{Err}}_{n,Q_{j+1}}(Q_j)\Vert .\end{array}$$

Simplifying terms we obtain

$$\begin{array}{l}\sum _{t=0}^T{L}^{(t+1)}E\left|{\overline{Q}}_t-Q_t\right|\le \sqrt{p}{M}_Q/\eta \sum _{t=0}^T{L}^{(t+1)}\sum _{j=t}^T{\left(Lp{M}_Q^2/\eta \right)}^{j-t}\Vert {\widetilde{Err}}_{n,Q_{j+1}}(Q_j)\Vert \\ +4p^{T+1/2}{M}_Q^{2T+1}{\eta }^{-(T+1)}{L}^{T+1}\underset{t}{max}\Vert {\widetilde{Err}}_{n,Q_{t+1}}(Q_t)-{\widetilde{Err}}_{Q_{t+1}}(Q_t)\Vert .\end{array}$$ (15)

Consider each component of each of the T +1, p dimensional vectors, ${\widetilde{Err}}_{n,Q_{i+1}}(Q_i))-{\widetilde{Err}}_{Q_{i+1}}(Q_i)$ for an ɛ′> 0:

$$P\left[\underset{i=0}{\overset{T}{\cup }}\underset{j=1}{\overset{p}{\cup }}\left\{\text{for\hspace{0.17em}}{\text{some}}_{{\theta }_i,{\theta }_{i+1}\in \mathrm{\Theta },q_i\in Q_i,q_{i+1}\in Q_{i+1}}\left|{\widetilde{Err}}_{n,Q_{i+1}}{(Q_i)}_j-{\widetilde{Err}}_{Q_{i+1}}(Q_i){)}_j\right|>{\varepsilon }^{\prime }\right\}\right].$$

This probability is in turn bounded above by

$$\sum _{i=0}^T\sum _{j=1}^{p}P\left[\text{for\hspace{0.17em}}{\text{some}}_{{\theta }_i,{\theta }_{i+1}\in \mathrm{\Theta },q_i\in Q_i,q_{i+1}\in Q_{i+1}}\left|{\widetilde{Err}}_{n,Q_{i+1}}{(Q_i)}_j-{\widetilde{Err}}_{Q_{i+1}}(Q_i){)}_j\right|>{\varepsilon }^{\prime }\right].$$

In Lemmas 17.2, 17.3, 17.5, Anthony and Bartlett (1999) provide an upper bound on the probability

$$P\left[\text{for\hspace{0.17em}some\hspace{0.17em}}f\in F\hspace{0.17em}\text{has\hspace{0.17em}}\left|E_n({\ell }_f)-E({\ell }_f)\right|\ge \hspace{0.17em}{\varepsilon }^{\prime }\right]$$

where ℓ_f (x, y) = (y − f (x))². These same lemmas (based on the classical arguments of symmetrization, permutation and reduction to a finite set) can be used for f ∈ F′ since the functions in F′ are uniformly bounded. Hence for each j = 1,..., p and t = 0,...,T

$$\begin{array}{l}P\left[\text{for\hspace{0.17em}some\hspace{0.17em}}{\theta }_t,{\theta }_{t+1}\in \mathrm{\Theta },q_t\in Q_t,q_{t+1}\in Q_{t+1}\hspace{0.17em}\text{has\hspace{0.17em}}\left|{\widetilde{Err}}_{n,Q_{t+1}}{(Q_t)}_j-{\widetilde{Err}}_{Q_{t+1}}{(Q_t)}_j\right|>{\varepsilon }^{\prime }\right]\\ \le 4N_1\left(\frac{{\varepsilon }^{\prime }}{\text{16}{M}^{\prime }},F^{\prime },2n\right)\hspace{0.17em}exp\hspace{0.17em}\{-\frac{{({\varepsilon }^{\prime }\text{)}}^{\text{2}}n}{32{(M^{\prime })}^2}\}.\end{array}$$

Set ɛ = p^T+1/2M^2T+1_Qη^−(T+1)L^T+1ɛ′. Thus for n satisfying

$$\begin{array}{l}4p(T+1)N_1\left(\frac{\varepsilon }{\text{16}{M}^{\prime }{p}^{T+1/2}{M}_Q^{2T+1}{\eta }^{-(T+1)}{L}^{T+1}},F^{\prime },2n\right)\\ exp\hspace{0.17em}\{-\frac{{\varepsilon }^{\text{2}}n}{\text{32(}{M}^{\prime }{)}^2{\left(p^{T+1/2}{M}_Q^{2T+1}{\eta }^{-(T+1)}{L}^{T+1}\right)}^2}\}\le \delta ,\end{array}$$ (16)

the first result of the theorem holds.

To simplify the constraint on n, we derive a covering number for F′ from covering numbers for the Q_t’s. Apply Lemma A2 part 1, to obtain

$$N_1(\varepsilon ,V_{t+1,n})\le N_1\left(\frac{\varepsilon }{\left|A\right|},Q_{t+1},\left|A\right|n\right)$$

for V_t+1 = {max_at+1Q_t+1(o_t+1, a_t+1) : Q_t+1 ∈ Q_t+1}. Next apply Lemma A2, parts 2 and 3, to obtain

$$\begin{array}{l}{N}_1(\varepsilon ,F^{\prime },n)\le \sum _{t=0}^{T-1}{N}_1\left(\frac{\varepsilon }{2\left|A\right|M^{\prime }},Q_{t+1},\left|A\right|n\right)N_1\left(\frac{\varepsilon }{2M^{\prime }},Q_t,n\right)\\ +N_1\left(\frac{\varepsilon }{M^{\prime }},Q_T,n\right).\end{array}$$

Theorems 11.6 and 18.4 of Anthony and Bartlett imply that $N_1(\varepsilon ,Q_t,n)\le e(p+1){\left(\frac{2e}{\varepsilon }\right)}^p$ for each t. Combining this upper bound with (16) and simplifying the algebra yields (14).

Next Lemma 2 implies:

$$\begin{array}{l}\int \left|V_{\tilde{\pi }}(o)-V_{\pi }(o)\right|dF(o)\hspace{0.17em}\le \hspace{0.17em}M\sum _{t=0}^{T}2L^{(t+1)}E\left|Q_t-{\overline{Q}}_t\right|\\ +M\sum _{t=0}^{T}2L^{(t+1)}E\left|{\overline{Q}}_t-{\tilde{Q}}_t\right|+M\sum _{t=0}^{T}2L^{(t+1)}E\left|{\tilde{Q}}_t-Q_{\tilde{\pi },t}\right|.\end{array}$$

This combined with the first result of the theorem implies the second result.

6. Discussion

Planning problems involving a single training set of trajectories are not unusual and can be expected to increase due to the widespread use of policies in the social and behavioral/medical sciences (see, for example, Rush et al., 2003; Altfeld and Walker, 2001; Brooner, and Kidorf, 2002); at this time these policies are formulated using expert opinion, clinical experience and/or theoretical models. However there is growing interest in formulating these policies using empirical studies (training sets). These training sets are collected under fixed exploration policies and thus while they allow exploration they do not allow exploitation, that is, online choice of the actions. If subjects are recruited into the study at a much slower rate than the calendar duration of the horizon, then it is possible to permit some exploitation; some of this occurs in the field of cancer research (Thall, Sung and Estey, 2002).

This paper considers the use of Q-learning with dynamic programming and function approximation for this planning purpose. However the mismatch between Q-learning and the goal of learning a policy that maximizes the value function has serious consequences and emphasizes the need to use all available science in choosing the approximation space. Often the available behaviorial or psychosocial theories provide qualitative information concerning the importance of different observations. In addition these theories are often represented graphically via directed acyclic graphs. However information at the level of the form of the conditional distributions connecting the nodes in the graphs is mostly unavailable. Also due to the complexity of the problems there are often unknown missing common causes of different nodes in the graphs. See http://neuromancer.eecs.umich.edu/dtr for more information and references. Methods that can use this qualitative information to minimize t! he mismatch are needed.

Appendix A

Recall that the distributions, P and P_π differ only with regards to the policy (see (1) and (2)). Thus the following result is unsurprising. Let f (O_T+1, A_T) be a (measurable) nonnegative function; then E_πf can be expressed in terms of an expectation with respect to the distribution P if we assume that p_t (a_t|o_t, a_t−1) > 0 for each (o_t, a_t ) pair and each t. The presence of the p _js in denominator below represent the price we pay because we only have access to training trajectories with distribution P; we do not have access to trajectories from distribution P_π.

Lemma A1 Assume that P_π[p₀(A₀|S₀) > 0] = 1 and P_π[p_t (A_t|O_t, A_t−1) > 0] = 1 for t = 1,...,T. For any (measurable) nonnegative function of g(O_t, A_t ), the P-probability that

$$E_{\pi }[g(O_t,A_t)|S_0]=E\left[\left(\prod _{\ell =0}^t\frac{1_{A_{\ell }={\pi }_{\ell }}}{p_{\ell }(A_{\ell }|O_{\ell },A_{\ell -1})}\right)g(O_t,A_t)|S_0\right]$$

is one for t = 0,...,T.

Proof: We need only prove that

$$E[h(S_0)E_{\pi }[g(O_t,A_t)|S_0]]=E\left[h(S_0)E\left[\left(\prod _{\ell =0}^t\frac{1_{A_{\ell }={\pi }_{\ell }}}{p_{\ell }(A_{\ell }|O_{\ell },A_{\ell -1})}\right)g(O_t,A_t)|S_0\right]\right]$$

for any (measurable) nonnegative function, h. Consider the two likelihoods ((1) and (2)) for a trajectory up to time t. Denote the dominating measure for the two likelihoods for the trajectory up to time t as λ_t. By assumption,

$$\begin{array}{l}\int h(s_0)g(o_t,a_t)\left(\prod _{\ell =0}^T\frac{1_{A_{\ell }={\pi }_{\ell }}}{p_{\ell }(a_{\ell }|o_{\ell },a_{\ell -1})}\right)f_0(s_0)p_0(a_0|s_0)\\ \prod _{j=1}^t{f}_j(s_j|o_{j-1},a_{j-1})p_j(a_j|o_j,a_{j-1})d{\lambda }_t(o_t,a_t)\\ =\int h(s_0)g(o_t,a_t)f_0(s_0)1_{a_0={\pi }_0(s_0)}\prod _{j=1}^t{f}_j(s_j|o_{j-1},a_{j-1})1_{a_j={\pi }_j(o_j,a_{j-1})}d{\lambda }_t(o_t,a_t).\end{array}$$

By definition the left hand side is $E\left[h(S_0)g(O_t,A_t)\left({\prod }_{\ell =0}^j\frac{1_{A_{\ell }={\pi }_{\ell }}}{p_{\ell }(A_{\ell }|O_{\ell },A_{\ell -1})}\right)\right]$ and the right hand side is E_π[h(S₀)g(O_t, A_t)]. Expressing both sides as the expectation of a conditional expectation, we obtain,

$$E_{\pi }[h(S_0)E_{\pi }[g(O_t,A_t)|S_0]]=E\left[h(S_0)E\left[g(O_t,A_t)\left(\prod _{\ell =0}^t\frac{1_{A_{\ell }={\pi }_{\ell }}}{p_{\ell }(A_{\ell }|O_{\ell },A_{\ell -1})}\right)|S_0\right]\right].$$

Note that the distribution of S₀ is the same regardless of how the actions are chosen, that is the distribution of S₀ is the same under both P and P_π. Thus

$$E[h(S_0)E_{\pi }[g(O_t,A_t)|S_0]]=E\left[h(S_0)E\left[g(O_t,A_t)\left(\prod _{\ell =0}^t\frac{1_{A_{\ell }={\pi }_{\ell }}}{p_{\ell }(A_{\ell }|A_{\ell },A_{\ell -1})}\right)|S_0\right]\right].$$

Lemma A2 For p, q, r, s, N positive integers and M_F, M_G, M_Θ positive reals, define the following classes of real valued functions,

$$\begin{array}{l}H\hspace{0.17em}\subseteq \hspace{0.17em}\left\{h(x,a)\hspace{0.17em}:\hspace{0.17em}x\hspace{0.17em}\in \hspace{0.17em}{ℝ}^p,\hspace{0.17em}a\hspace{0.17em}\in \hspace{0.17em}\{1,\dots ,N\}\right\}\\ F\hspace{0.17em}\subseteq \hspace{0.17em}\left\{f(x)\hspace{0.17em}:\hspace{0.17em}x\hspace{0.17em}\in \hspace{0.17em}{ℝ}^q,\underset{x}{sup}\left|f(x)\right|\hspace{0.17em}\le \hspace{0.17em}{M}_F\right\}\\ G\hspace{0.17em}\subseteq \hspace{0.17em}\left\{g(x,y)\hspace{0.17em}:\hspace{0.17em}x\hspace{0.17em}\in \hspace{0.17em}{ℝ}^q,y\hspace{0.17em}\in \hspace{0.17em}{ℝ}^r\underset{x,y}{sup}\left|g(x,y)\right|\hspace{0.17em}\le \hspace{0.17em}{M}_G\right\}\end{array}$$

and

$$\mathrm{\Theta }\hspace{0.17em}\subseteq \hspace{0.17em}\left\{\theta \in \hspace{0.17em}\hspace{0.17em}{ℝ}^s:\hspace{0.17em}\underset{i=1,\dots ,s}{max}\left|{\theta }_i\right|\le \hspace{0.17em}{M}_{\mathrm{\Theta }}\right\}.$$

The following hold.

1. If V = {max_a h(x, a) : h ∈ H}then N₁ (ɛ, V n) ≤ N₁ (ɛ/N, H Nn).

2. For |a| ≤ 1, |b| ≤ 1, if V = {a f (x) + bg(x, y) : f ∈ F, g ∈ G} then N₁(ɛ, V, n) ≤ N₁(ɛ/2, F, n)N₁(ɛ/2, G, n).

3. If V = F ∪ G then N₁(ɛ,V, n) ≤ N₁(ɛ,F, n) + N₁(ɛ, G, n).

4. If V = {θ₁ f₁(x) + ... + θ_s f_s(x) : f_i ∈ F, (θ₁,..., θ_s) ∈ Θ} then $N_1(\varepsilon ,V,n)\hspace{0.17em}\le \hspace{0.17em}e(s+1)\hspace{0.17em}{\left(\frac{4es{M}_{\mathrm{\Theta }}{M}_F}{\varepsilon }\right)}^s\hspace{0.17em}{N}_{\infty }{\left(\frac{\varepsilon }{4s{M}_{\mathrm{\Theta }}},F,n\right)}^s.$

Proof. We prove 1. and 4.; the proofs of 2. and 3. are straightforward and are omitted. Consider 1. Given (x₁,..., x_n), the ɛ-covering number for the class of points in ℝ^Nn, {(h(x_i, a) : i = 1,...,n, a = 1,...N); h ∈ H} is bounded above by N₁ (ɛ, H, Nn). Note that for (z_ia, i = 1,...,n, a = 1,...,N),

$$\begin{array}{l}1/n\hspace{0.17em}\sum _{i=1}^n\left|\underset{a=1,\dots ,N}{max}\hspace{0.17em}h(x_i,a)-\underset{a=1,\dots ,N}{max}\hspace{0.17em}{z}_{ia}\right|\le \hspace{0.17em}1/n\hspace{0.17em}\sum _{i=1}^n\underset{a=1,\dots ,N}{max}\left|\hspace{0.17em}h(x_i,a)-\hspace{0.17em}{z}_{ia}\right|\\ \le \hspace{0.17em}1/n\hspace{0.17em}\sum _{i=1}^n\sum _{a=1}^N\left|\hspace{0.17em}h(x_i,a)-\hspace{0.17em}{z}_{ia}\right|.\end{array}$$

Thus the ɛ-covering number for the class of points in ℝⁿ, ${ℝ}^n,\left\{({max}_{a=1}^{N}h(x_i,a)\hspace{0.17em}:\hspace{0.17em}i=1,\dots ,n)\hspace{0.17em};\hspace{0.17em}h\hspace{0.17em}\in \hspace{0.17em}H\right\}$ is bounded above by N₁ (ɛ, H, Nn). Using the definition of covering numbers for classes of functions we obtain $N_1(\varepsilon ,V,n)\le N_1\left(\frac{\varepsilon }{N},H,Nn\right).$

Next consider 4. Put x = (x₁,..., x_n) (each x_i ∈ ℝ^q) and f (x_i) = ( f₁(x_i),..., f_s(x_i))^T. Then there exists {z₁,..., z_N}, (N = N_∞ (ɛ/(4sM_Θ), F, n); z_j ∞ ℝⁿ) that form the centers of an ɛ/(4sM_Θ)-cover for F. To each z_j we can associate an f ∈ F, say f ^*_j so that {f ^*₁,...,f ^*_N} form the centers of an ɛ/(2sM_Θ)-cover for F. Then given {f₁,..., f_s} ∈ F there exists j^* ∈ {1,...,N } for j = 1,...,s, so that max_1≤j_≤s max_1≤i_≤n | f_j(x_i) − f^*_j*(x_i)| ≤ ɛ /( 2sM_Θ). Then

$$(1/n)\hspace{0.17em}\sum _{i=1}^n|\sum _{j=1}^s{\theta }_j{f}_j(x_i)-{\theta }_j{f}_{j*}^{*}(x_i)|\le \hspace{0.17em}\varepsilon /\text{2}.$$

Define $F^{\prime }=\left\{{\sum }_{j=1}^s{\theta }_j{f}_{j*}^{*}\hspace{0.17em}:{\theta }_j\hspace{0.17em}\in \hspace{0.17em}\mathrm{\Theta }\right\}.$ Theorems 11.6 and 18.4 of Anthony and Bartlett (1996) imply that $N_1(\varepsilon /\text{2},F^{\prime },n)\hspace{0.17em}\le \hspace{0.17em}e(s+1)\hspace{0.17em}{\left(\frac{4es{M}_{\mathrm{\Theta }}{M}_F}{\varepsilon }\right)}^s$ These two combine to yield the result.