Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning

Abstract

The vision for precision medicine is to use individual patient characteristics to inform a personalized treatment plan that leads to the best possible health­care for each patient. Mobile technologies have an important role to play in this vision as they offer a means to monitor a patient’s health status in real-time and subsequently to deliver interventions if, when, and in the dose that they are needed. Dynamic treatment regimes formalize individualized treatment plans as sequences of decision rules, one per stage of clinical intervention, that map current patient information to a recommended treatment. However, most existing methods for estimating optimal dynamic treatment regimes are designed for a small number of fixed decision points occurring on a coarse time-scale. We propose a new reinforcement learning method for estimating an optimal treatment regime that is applicable to data collected using mobile technologies in an out­patient setting. The proposed method accommodates an indefinite time horizon and minute-by-minute decision making that are common in mobile health applications. We show that the proposed estimators are consistent and asymptotically normal under mild conditions. The proposed methods are applied to estimate an optimal dynamic treatment regime for controlling blood glucose levels in patients with type 1 diabetes.

1. Introduction

The use of mobile devices in clinical care, called mobile health (mHealth), provides an effective and scalable platform to assist patients in managing their illness (Free et al., 2013; Steinhubl et al., 2013). Advantages of mHealth interventions include real-time communication between a patient and their health-care provider as well as systems for delivering training, teaching, and social support (Kumar et al., 2013). Mobile technologies can also be used to collect rich longitudinal data to estimate optimal dynamic treatment regimes and to deliver treatment that is deeply tailored to each individual patient. We propose a new estimator of an optimal treatment regime that is suitable for use with longitudinal data collected in mHealth applications.

A dynamic treatment regime provides a framework to administer individualized treatment over time through a series of decision rules. Dynamic treatment regimes have been well-studied in the statistical and biomedical literature (Murphy, 2003; Robins, 2004; Moodie et al., 2007; Kosorok and Moodie, 2015; Chakraborty and Moodie, 2013) and furthermore, statistical considerations in mHealth have been studied by, for example, Liao et al. (2016) and Klasnja et al. (2015). Although mobile technology has been successfully utilized in clinical areas such as diabetes (Quinn et al., 2011; Maahs et al., 2012), smoking cessation (Ali et al., 2012), and obesity (Bexelius et al., 2010), mHealth poses some unique challenges that preclude direct application of existing methodologies for dynamic treatment regimes. For example, mHealth applications typically have no definite time horizon in the sense that treatment decisions are made continually throughout the life of the patient with no fixed time point for the final treatment decision; estimation of an optimal treatment strategy must utilize data collected over a much shorter time period than that over which treatment would be applied in practice; the momentary signal may be weak and may not directly measure the outcome of interest; and estimation of optimal treatment strategies must be done online as data accumulate.

This work is motivated in part by our involvement in a study of mHealth as a management tool for type 1 diabetes. Type 1 diabetes is an autoimmune disease wherein the pancreas produces insufficient levels of insulin, a hormone needed to regulate blood glucose concentration. Patients with type 1 diabetes are continually engaged in management activities including monitoring glucose levels, timing and dosing insulin injections, and regulating diet and physical activity. Increased glucose monitoring and attention to self-management facilitate more frequent treatment adjustments and have been shown to improve patient outcomes (Levine et al., 2001; Haller et al., 2004; Ziegler et al., 2011). Thus, patient outcomes have the potential to be improved by diabetes management tools which are deeply tailored to the continually evolving health status of each patient. Mobile technologies can be used to collect data on physical activity, glucose, and insulin at a fine granularity in an outpatient setting (Maahs et al., 2012). There is great potential for using these data to create comprehensive and accessible mHealth interventions for clinical use. We envision application of this work for use before the artificial pancreas (Weinzimer et al., 2008; Kowalski, 2015; Bergenstal et al., 2016) becomes widely available.

In our motivating example as well as other mHealth applications, the goal is to treat a chronic disease over the long term. However, data will typically be collected over a short time period. Because the time frame of data collection is much shorter than the time frame of application, standard methods for longitudinal data analysis, such as generalized estimating equations or mixed models, cannot be used. We assume that the data collected in the field consist of a sample from a stationary Markov process, which allows us to estimate a dynamic treatment regime that will lead to good outcomes over the long term using data collected over a much shorter time period.

The sequential decision making process can be modeled as a Markov decision process (Puterman, 2014) and the optimal treatment regime can be estimated using reinforcement learning algorithms such as Q-learning (Murphy, 2005; Zhao et al., 2009; Tang and Kosorok, 2012; Schulte et al., 2014). Ertefaie (2014) proposed a variant of greedy gradient Q-learning (GGQ) to estimate optimal dynamic treatment regimes in infinite horizon settings (see also Maei et al., 2010). In GGQ, the form of the estimated Q-function dictates the form of the estimated optimal treatment regime. Thus, one must choose between a parsimonious model for the Q-function at the risk of model misspecification or a complex Q-function that yields unintelligible treatment regimes. Furthermore, GGQ requires modeling a non-smooth function of the data, which creates complications (Laber et al., 2014; Linn et al., 2017). Applications of mHealth require methods that can both estimate a policy from a fixed sample of retrospective data (offline estimation) and estimate a policy that is updated as data accumulate (online estimation). Online estimation has been given considerable attention in the field of reinforcement learning, particularly in engineering applications (Doya, 2000; Kober and Peters, 2012), with special consideration given to algorithms that provide fast updates in situations where new data accumulate multiple times per second. In our mHealth examples, treatment decisions may be made multiple times per day or even every hour or minute; however, rapid updating of the estimated policy at the scale of engineering applications is not needed. Therefore, we are able to focus on different aspects of the problem without needing to ensure very fast estimation. We propose an alternative estimation method for infinite horizon dynamic treatment regimes that is suited to mHealth applications. Our approach, which we call V-learning, involves estimating the optimal policy among a prespecified class of policies (Zhang et al., 2012, 2013). It requires minimal assumptions about the data-generating process and permits estimating a randomized decision rule that can be implemented online as data accumulate.

In Section 2, we describe the setup and present our method for offline estimation using data from a micro-randomized trial or observational study. In Section 3, we extend our method for application to online estimation with accumulating data. Theoretical results, including consistency and asymptotic normality of the proposed estimators, are presented in Section 4. We compare the proposed method to GGQ using simulated data in Section 5. A case study using data from patients with type 1 diabetes is presented in Section 6 and we conclude with a discussion in Section 7. Proofs of technical results are in the Appendix.

2. Offline estimation from observational data

We assume that the available data are ${\left\{\left(S_i^1,A_i^1,S_i^2,\dots ,S_i^{T_i},A_i^{T_i},S_i^{T_i+1}\right)\right\}}_{i=1}^n$, which comprise n independent, identically distributed trajectories (S¹, A¹, S², …, S^T, A^T, S^T+1), where: $S^t\in {ℝ}^p$ denotes a summary of patient information collected up to and including time t; $A^t\in \mathcal{A}$ denotes the treatment assigned at time t; and $T\in {\mathbb{Z}}_{+}$ denotes the (possibly random) patient follow-up time. In the motivating example of type 1 diabetes, S^t could contain a patient’s blood glucose, dietary intake, and physical activity in the hour leading up to time t and A^t could denote an indicator that an insulin injection is taken at time t. We assume that the data-generating model is a time-homogeneous Markov process so that S^t+1 ╨ (A^t−1, S^t+1, …, A¹, S¹)|(A^t, S^t) and the conditional density p(s^t+1|a^t, s^t) is the same for all t ≥ 1. Let L^t ∈ {0, 1} denote an indicator that the patient is still in follow-up at time t, i.e., L^t = 1 if the patient is being followed at time t and zero otherwise. We assume that L^t is contained in S^t so that P(L^t+1 = 1| A^t, S^t,…, A¹, S¹) = P(L^t+1 = 1|A^t, S^t). It is not necessary for time points to be evenly spaced or homogeneous across patients. For example, we can define the decision times as those times at which data are observed and include time since the previous observation in S^t. Defining the decision times in this way allows us to effectively handle intermittent missing data, and thus we can assume that L^t = 0 implies L^t+1 = 0 with probability one. Furthermore, we assume a known utility function $u:{ℝ}^p\times \mathcal{A}\times {ℝ}^p\to ℝ$ so that U^t = u (S^t+1, A^t, S^t) measures the ‘goodness’ of choosing treatment A^t in state S^t and subsequently transitioning to state S^t+1. In our motivating example, the utility at time t could be a measure of how infrequently the patient’s blood glucose concentration deviates from the optimal range over the hour preceding and following time t. The goal is to select treatments to maximize expected cumulative utility; treatment selection is formalized using a treatment regime (Schulte et al., 2014; Kosorok and Moodie, 2015) and the utility associated with any regime is defined using potential outcomes (Rubin, 1978).

Let $\mathcal{B}\left(\mathcal{A}\right)$ denote the space of probability distributions over $\mathcal{A}$. A treatment regime in this context is a function $\pi :\text{dom}{S}^t\to \mathcal{B}\left(\mathcal{A}\right)$ so that, under π, a decision maker presented with state S^t= s^t at time t will select action $a^t\in \mathcal{A}$ with probability π(a^t; s^t). Define ${\overline{a}}^t=\left(a^1,\dots ,a^t\right)\in {\mathcal{A}}^t$, and ${\overline{a}}^{\infty }=\left(a^1,a^2,\dots \right)\in {\mathcal{A}}^{\infty }$. The set of potential outcomes is

$$W^{\ast }=\left\{S^1,S^{\ast 2}\left(a^1\right),\dots ,S^{\ast T^{\ast }\left({\overline{a}}^{\infty }\right)}\left({\overline{a}}^{T^{\ast }\left({\overline{a}}^{\infty }\right)-1}\right):T^{\ast }\left({\overline{a}}^{\infty }\right)=\text{inf}\left\{t\ge 1:L^{\ast t}\left({\overline{a}}^{t-1}\right)=0\right\},{\overline{a}}^{\infty }\in {\mathcal{A}}^{\infty }\right\},$$

where $S^{\ast t}\left({\overline{a}}^{t-1}\right)$ is the potential state and $L^{\ast t}\left({\overline{a}}^{t-1}\right)$ is the potential follow-up status at time t under treatment sequence ${\overline{a}}^{t-1}$. Thus, the potential utility at time t is $U^{\ast t}\left({\overline{a}}^t\right)=u\left\{S^{\ast \left(t+1\right)}\left({\overline{a}}^t\right),a^t,S^{\ast t}\left({\overline{a}}^{t-1}\right)\right\}$. For any π, define ${\left\{{\xi }_{\pi }^t(\cdot )\right\}}_{t\ge 1}$ to be a sequence of independent, $\mathcal{A}$ −valued stochastic processes indexed by dom S^t such that $P\left\{{\xi }_{\pi }^t\left(s^t\right)=a^t\right\}=\pi \left(a^t;s^t\right)$. The potential follow-up time under π is

$$T^{\ast }\left(\pi \right){\displaystyle \sum _{t\ge 1}{\displaystyle \sum _{{\overline{a}}^t\in {\mathcal{A}}^t}t1\left\{{\text{sup}}_{{\underset{\_}{a}}^{t+1}}{T}^{\ast }\left({\overline{a}}^t,{\underset{\_}{a}}^{t+1}\right)=t\right\}}}{\displaystyle \prod _{\upsilon =1}^{t}1\left[{\xi }_{\pi }^{\upsilon }\left\{S^{\ast \upsilon }\left({\overline{a}}^{\upsilon -1}\right)\right\}=a^{\upsilon }\right]},$$

where ${\underset{\_}{a}}^{t+1}=\left(a^{t+1},a^{t+2},\dots \right)$. The potential utility under π at time t is

$$U^{\ast t}\left(\pi \right)=\{\begin{array}{ll}{\displaystyle {\sum }_{{\overline{a}}^t\in {\mathcal{A}}^t}{U}^{\ast t}\left({\overline{a}}^t\right){\displaystyle {\prod }_{\upsilon =1}^{t}1\left[{\xi }_{\pi }^{\upsilon }\left\{S^{\ast \upsilon }\left({\overline{a}}^{\upsilon -1}\right)\right\}=a^{\upsilon }\right],}}\hfill & \text{if}{T}^{\ast }\left(\pi \right)\ge t\hfill \\ 0,\hfill & \text{otherwise},\hfill \end{array}$$

where $S^{\ast 1}\left({\overline{a}}^0\right)=S^1$. Thus, utility is set to zero after a patient is lost to follow-up. However, in certain situations, utility may be constructed so as to take a negative value at the time point when the patient is lost to follow-up, e.g., if the patient discontinues treatment because of a negative effect associated with the intervention. Define the state-value function $V\left(\pi ,s^t\right)=\mathbb{E}\left\{{\displaystyle {\sum }_{k\ge 0}{\gamma }^k{U}^{\ast \left(t+k\right)}\left(\pi \right)}|S^t=s^t\right\}$ (Sutton and Barto, 1998), where γ ∈ (0,1) is a fixed constant that captures the trade-off between short- and long-term outcomes. For any distribution $\mathcal{R}$ on dom S¹, define the value function with respect to reference distribution $\mathcal{R}$ as $V_{\mathcal{R}}\left(\pi \right)=\int V\left(\pi ,s\right)\text{d}\mathcal{R}\left(s\right)$; throughout, we assume that this reference distribution is fixed. The reference distribution can be thought of as a distribution of initial states and we estimate it from the data in the implementation in Sections 5 and 6. For a prespecified class of regimes, Π, the optimal regime, ${\pi }_{\mathcal{R}}^{\text{opt}}\in \Pi$, satisfies $V_{\mathcal{R}}\left({\pi }_{\mathcal{R}}^{\text{opt}}\right)\ge V_{\mathcal{R}}\left(\pi \right)$ for all π ∈ Π. The goal is to estimate ${\pi }_{\mathcal{R}}^{\text{opt}}$ using data collected from n patients, where patient i is followed for T_i time points, i = 1, …, n. Thus, T_i represents the number of treatment decisions made for patient i in the observed data; however, because the observed data are assumed to come from a time-homogeneous Markov chain, the estimated policy could be applied in a population indefinitely.

To construct an estimator of ${\pi }_{\mathcal{R}}^{\text{opt}}$, we make a series of assumptions that connect the potential outcomes in W^* with the data-generating model.

Assumption 1. Strong ignorability, A^t ╨ W^* | S^t for all t.

Assumption 2. Consistency, $S^t=S^{\ast t}\left({\overline{A}}^{t-1}\right)$ for all t and $T=T^{\ast }\left({\overline{A}}^{\infty }\right)$.

Assumption 3. Positivity, there exists c₀ > 0 so that P(A^t = a^t|S^t = s^t) ≥ c₀ for all $a^t\in \mathcal{A},s^t\in \text{dom }{S}^t$, and all t.

In addition, we implicitly assume that there is no interference among the experimental units. These assumptions are common in the context of estimating dynamic treatment regimes (Robins, 2004; Hernan and Robins, 2010; Schulte et al., 2014). Assumption 1 implies that there are no unmeasured confounders and assumptions 1 and 3 hold by construction in a micro-randomized trial (Klasnja et al., 2015; Liao et al., 2016).

Let μ^t(a^t; s^t) = P(A^t = a^t|S^t = s^t) for each t ≥ 1. In a micro-randomized trial, μ^t(a^t; s^t) is a known randomization probability; in an observational study, it must be estimated from the data. The following lemma characterizes $V_{\mathcal{R}}\left(\pi \right)$ for any regime, π, in terms of the data-generating model (see also Lemma 4.1 of Murphy et al., 2001). A proof is provided in the appendix.

Lemma 2.1. Let π denote an arbitrary regime and γ ∈ (0, 1) a discount factor. Then, under assumptions 1–3 and provided interchange of the sum and integration is justified, the state-value function of π at s^t is

$$V\left(\pi ,s^t\right)={\displaystyle \sum _{k\ge 0}\mathbb{E}\left[{\gamma }^k{U}^{t+k}\left\{{\displaystyle \prod _{\upsilon =0}^k\frac{\pi \left(A^{\upsilon +t};S^{\upsilon +t}\right)}{{\mu }^{\upsilon +t}\left(A^{\upsilon +t};S^{\upsilon +t}\right)}}\right\}|S^t=s^t\right]}.$$ (1)

The preceding result will form the basis for an estimating equation for $V_{\mathcal{R}}\left(\pi \right)$. Write the right hand side of (1) as

$$\begin{array}{lll}V\left(\pi ,S^t\right)\hfill & =\hfill & \mathbb{E}\left\{\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}\left(U^t+\gamma {\displaystyle \sum _{k\ge 0}\mathbb{E}\left[{\gamma }^k{U}^{t+k+1}\left\{{\displaystyle \prod _{\upsilon =0}^k\frac{\pi \left(A^{\upsilon +t+1};S^{\upsilon +t+1}\right)}{{\mu }^{\upsilon +t+1}\left(A^{\upsilon +t+1};S^{\upsilon +t+1}\right)}}\right\}|S^{t+1}\right]}\right)|S^t\right\}\hfill \\ \hfill & =\hfill & \mathbb{E}\left[\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}\left\{U^t+\gamma V\left(\pi ,S^{t+1}\right)\right\}|S^t\right],\hfill \end{array}$$

from which it follows that

$$0=\mathbb{E}\left[\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}\left\{U^t+\gamma V\left(\pi ,S^{t+1}\right)-V\left(\pi ,S^t\right)\right\}|S^t\right].$$

Subsequently, for any function ψ defined on dom S^t, the state-value function satisfies

$$0=\mathbb{E}\left[\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}\left\{U^t+\gamma V\left(\pi ,S^{t+1}\right)-V\left(\pi ,S^t\right)\right\}\psi \left(S^t\right)\right],$$ (2)

which is an importance-weighted variant of the well-known Bellman optimality equation (Sutton and Barto, 1998).

Let V(π, s; θ^π) denote a model for V(π, s) indexed by ${\theta }^{\pi }\in \Theta \subseteq {ℝ}^q$. We assume that the map θ^π ⟼ V(π, s;θ^π) is differentiable everywhere for each fixed s and π. Let ${\nabla }_{{\theta }^{\pi }}V\left(\pi ,s;{\theta }^{\pi }\right)$ denote the gradient of V(π, s; θ^π) and define

$${\Lambda }_n\left(\pi ,{\theta }^{\pi }\right)=\frac{1}{n}{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{t=1}^{T_i}\frac{\pi \left(A_i^t;S_i^t\right)}{{\mu }^t\left(A_i^t;S_i^t\right)}}}\left\{U_i^t+\gamma V\left(\pi ,S_i^{t+1};{\theta }^{\pi }\right)-V\left(\pi ,S_i^t;{\theta }^{\pi }\right)\right\}{\nabla }_{{\theta }^{\pi }}V\left(\pi ,S_i^t;{\theta }^{\pi }\right).$$ (3)

Given a positive definite matrix $\Omega \in {ℝ}^{q\times q}$ and penalty function $\mathcal{P}:{ℝ}^q\to {ℝ}_{+}$, define ${\widehat{\theta }}_n^{\pi }=\text{arg}{\text{min}}_{{\theta }^{\pi }\in \Theta }\left\{{\Lambda }_n{\left(\pi ,{\theta }^{\pi }\right)}^{⊺}\Omega {\Lambda }_n\left(\pi ,{\theta }^{\pi }\right)+{\lambda }_n\mathcal{P}\left({\theta }^{\pi }\right)\right\}$, where λ_n is a tuning parameter. Subsequently, $V\left(\pi ,s;{\widehat{\theta }}_n^{\pi }\right)$ is the estimated state-value function under π in state s. Thus, given a reference distribution, $\mathcal{R}$, the estimated value of a regime, π, is ${\widehat{V}}_{n,\mathcal{R}}\left(\pi \right)={\displaystyle \int V\left(\pi ,s;{\widehat{\theta }}_n^{\pi }\right)}\text{d}\mathcal{R}\left(S\right)$ and the estimated optimal regime is ${\widehat{\pi }}_n=\text{arg}{\text{max}}_{\pi \in \Pi }{\widehat{V}}_{n,\mathcal{R}}\left(\pi \right)$. The idea of V-learning is to use estimating equation (3) to estimate the value of any policy and maximize estimated value over a class of policies; we will discuss strategies for this maximization below.

V-learning requires a parametric class of policies. Assuming that there are K possible treatments, a₁, … , a_K, we can define a parametric class of policies as follows. Define $\pi \left(a_j;s,\beta \right)=\text{exp}\left(s^{⊺}{\beta }_j\right)/\left\{1+{\displaystyle {\sum }_{k-1}^{K-1}\text{exp}\left(s^{⊺}{\beta }_k\right)}\right\}$ for j = 1, … , K − 1, and $\pi \left(a_K;s\right)=1/\left\{1+{\displaystyle {\sum }_{k=1}^{K-1}\text{exp}\left(s^{⊺}{\beta }_k\right)}\right\}$. This defines a class of randomized policies, Π, parametrized by $\beta ={\left({\beta }_1^{⊺},\dots ,{\beta }_{K-1}^{⊺}\right)}^{⊺}$, where β_k is a vector of parameters for the k-th treatment. Under a policy in this class defined by β, actions are selected stochastically according to the probabilities π(a_j; s, β), j = 1, … , K. In the case of a binary treatment, a policy in this class reduces to π(1; s, β) = exp(s^⊺β)/ {1 + exp(s^⊺β)} and π(0; s, β) = 1/ {1 + exp(s^⊺β)} for a p × 1 vector β. This class of policies is used in the implementation in Sections 5 and 6.

V-learning also requires a class of models for the state value function indexed by a parameter, θ^π. We use a basis function approximation (Hastie et al., 2009; Long et al., 2010). Let Φ = (ϕ₁, … , ϕ_q)^⊺ be a vector of prespecified basis functions and let $\Phi \left(s_i^t\right)={\left\{{\varphi }_1\left(s_i^t\right),\dots ,{\varphi }_q\left(s_i^t\right)\right\}}^{⊺}.$ Let $V\left(\pi ,s_i^t;{\theta }^{\pi }\right)=\Phi {\left(s_i^t\right)}^{⊺}{\theta }^{\pi }$ Under this working model,

$$\begin{array}{ll}{\Lambda }_n\left(\pi ,{\theta }^{\pi }\right)\hfill & =\left[n^{-1}{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{t=1}^{T_i}\frac{\pi \left(A_i^t;S_i^t\right)}{{\mu }^t\left(A_i^t;S_i^t\right)}\left\{\gamma \Phi \left(S_i^t\right)\Phi {\left(S_i^{t+1}\right)}^{⊺}-\Phi \left(S_i^t\right)\Phi {\left(S_i^t\right)}^{⊺}\right\}}}\right]{\theta }^{\pi }\hfill \\ \hfill & +n^{-1}{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{t=1}^{T_i}\left\{\frac{\pi \left(A_i^t;S_i^t\right)}{{\mu }^t\left(A_i^t;S_i^t\right)}{U}_i^t\Phi \left(S_i^t\right)\right\}.}}\hfill \end{array}$$ (4)

Computational efficiency is gained from the linearity of $V\left(\pi ,s_i^t;{\theta }^{\pi }\right)$ in θ^π; flexibility can be achieved through the choice of Φ. We recommend Gaussian basis functions as they offer the highest degree of flexibility. However, along with Gaussian basis functions, we also examine the performance of V-learning using linear and polynomial basis functions in Sections 5 and 6 as these offer reasonable alternatives.

The algorithm for V-learning is given in Algorithm 1 below. The algorithm can be terminated when $\Vert {\beta }^k-{\beta }^{k-1}\Vert$ is below some small threshold. The update in step can be achieved with a variety of existing optimization methods. In our implementation, we use the BFGS algorithm (Dai, 2002) as implemented in the optim function in R software (R Core Team, 2016). Because the objective function is not necessarily convex, care must be taken when selecting the starting point, β¹ (step 2). In our implementation, we use simulated annealing as implemented in the optim function in R to find an appropriate starting point.

Algorithm 1: V-learning.
1 Initialize a class of policies, $\Pi =\left\{{\pi }_{\beta }:\beta \in \mathcal{B}\right\}$, and a model, V (π, s; θπ);
2 Set k = 1 and initialize β1 to a starting value in $\mathcal{B}$;
3 while Not converged do
4  Estimate ${\widehat{\theta }}_n^{{\pi }_{{\beta }^k}}=\text{arg}{\text{min}}_{{\theta }^{{\pi }_{{\beta }^k}}\in \Theta }\left\{{\Lambda }_n{\left({\pi }_{{\beta }^k},{\theta }^{{\pi }_{{\beta }^k}}\right)}^{⊺}\Omega {\Lambda }_n\left({\pi }_{{\beta }^k},{\theta }^{{\pi }_{{\beta }^k}}\right)+{\lambda }_n\mathcal{P}\left({\theta }^{{\pi }_{{\beta }^k}}\right)\right\}$;
5  Evaluate ${\widehat{V}}_{n,\mathcal{R}}\left({\pi }_{{\beta }^k}\right)={\displaystyle \int V\left({\pi }_{{\beta }^k},S;{\widehat{\theta }}_n^{{\pi }_{{\beta }^k}}\right)}\text{d}\mathcal{R}\left(s\right)$;
6  Set ${\beta }^{k+1}={\beta }^k+{\alpha }^k{\nabla }_{{\beta }_k}{\widehat{V}}_{n,\mathcal{R}}\left({\pi }_{{\beta }^k}\right)$ for some step size, αk, where ${\nabla }_{{\beta }_k}{\widehat{V}}_{n,\mathcal{R}}\left({\pi }_{{\beta }^k}\right)$ is the gradient of ${\widehat{V}}_{n,\mathcal{R}}\left({\pi }_{{\beta }^k}\right)$;
7 end

2.1. Greedy gradient Q-learning

Here we briefly discuss an existing method for infinite horizon dynamic treatment regimes, which will be used for comparison in the simulation studies in Section 5.

Ertefaie (2014) introduced greedy gradient Q-learning (GGQ) for estimating dynamic treatment regimes in infinite horizon settings (see also Maei et al., 2010; Murphy et al., 2016).

Define $Q^{\pi }\left(s^t,a^t\right)=\mathbb{E}\left\{{{\displaystyle {\sum }_{k\ge 0}{\gamma }^{k}U}}^{t+k}\left(\pi \right)|S^t=s^t,A^t=a^t\right\}$. The Bellman optimality equation (Sutton and Barto, 1998) is

$$Q^{\text{opt}}\left(s^t,a^t\right)=\mathbb{E}\left\{U^t+\gamma \underset{a\in \mathcal{A}}{\text{max}}{Q}^{\text{opt}}\left(S^{t+1},a\right)|S^t=s^t,A^t=a^t\right\}.$$ (5)

Let Q(s, a; η^opt) be a parametric model for Q^opt(s, a) indexed by ${\eta }^{\text{opt}}\in H\subseteq {ℝ}^q$. In our implementation, we model Q(s, a; η^opt) as a linear function with interactions between all state variables and treatment. The Bellman optimality equation motivates the estimating equation

$$D_n\left({\eta }^{\text{opt}}\right)=\frac{1}{n}{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{t=1}^{T_i}\left\{U_i^t+\gamma \underset{a\in \mathcal{A}}{\text{max}}Q\left(S_i^{t+1},a;{\eta }^{\text{opt}}\right)-Q\left(S_i^t,A_i^t;{\eta }^{\text{opt}}\right)\right\}}}{\nabla }_{{\eta }^{\text{opt}}}Q\left(S_i^t,A_i^t;{\eta }^{\text{opt}}\right).$$ (6)

For a positive definite matrix, Ω, we estimate η^opt using ${\widehat{\eta }}_n^{\text{opt}}=\text{arg}{\text{min}}_{\eta \in H}{D}_n{\left(\eta \right)}^{⊺}\Omega D_n\left(\eta \right)$. The estimated optimal policy in state s selects action ${\widehat{\pi }}_n\left(s\right)={\text{max}}_{a\in \mathcal{A}}Q\left(s,a;{\widehat{\eta }}_n^{\text{opt}}\right)$. This optimization problem is non-convex and non-differentiable in η^opt. However, it can be solved with a generalization of the greedy gradient Q-learning algorithm of Maei et al. (2010), and hence is referred to as GGQ by Ertefaie (2014) and in the following.

The performance of GGQ has been demonstrated in the context of chronic diseases with large sample sizes and a moderate number of time points. However, in mHealth applications, it is common to have small sample sizes and a large number of time points, with decisions occurring at a fine granularity. In GGQ, the estimated policy depends directly ${\widehat{\eta }}_n^{\text{opt}}$ and, therefore, depends on modeling the transition probabilities of the data-generating process. Furthermore, estimating equation (6) contains a non-smooth max operator, which makes estimation difficult without large amounts of data (Laber et al., 2014; Linn et al., 2017). V-learning only requires modeling the policy and the value function rather than the data-generating process and directly maximizes estimated value over a class of policies, thereby avoiding the non-smooth max operator in the estimating equation (compare equations (3) and (6)); these attributes may prove advantageous in mHealth settings.

3. Online estimation from accumulating data

Suppose we have accumulating data ${\left\{\left(S_i^1,A_i^1,S_i^2,\dots \right)\right\}}_{i=1}^n$, where $S_i^t$ and $A_i^t$ represent the state and action for patient i = 1, … , n at time t ≥ 1. At each time t, we estimate an optimal policy in a class, Π, using data collected up to time t, take actions according to the estimated optimal policy, and estimate a new policy using the resulting states. Let ${\widehat{\pi }}_n^t$ be the estimated policy at time t, i.e., ${\widehat{\pi }}_n^t$ is estimated after observing state S^t+1 and before taking action A^t+1. If Π is a class of randomized policies, we can select an action for a patient presenting with S^t+1 = s^t+1 according to ${\widehat{\pi }}_n^t\left(\cdot ;s^{t+1}\right)$, i.e., we draw A^t+1 according to the distribution $P\left(A^{t+1}=a\right)={\widehat{\pi }}_n^t\left(\alpha ;s^{t+1}\right)$. If a class of deterministic policies is of interest, we can inject some randomness into ${\widehat{\pi }}_n^t$ to facilitate exploration. One way to do this is an ϵ-greedy strategy (Sutton and Barto, 1998), which selects the estimated optimal action with probability 1 – ϵ and otherwise samples equally from all other actions. Because an ϵ-greedy strategy can be used to introduce randomness into a deterministic policy, we can assume a class of randomized policies.

At each time t ≥ 1, let ${\widehat{\theta }}_{n,t}^{\pi }=\text{arg}{\text{min}}_{{\theta }^{\pi }\in \Theta }\left\{{\Lambda }_{n,t}{\left(\pi ,{\theta }^{\pi }\right)}^{⊺}\Omega {\Lambda }_{n,t}\left(\pi ,{\theta }^{\pi }\right)+{\lambda }_n\mathcal{P}\left({\theta }^{\pi }\right)\right\}$ where Ω, λ_n, and $\mathcal{P}$ are as defined in Section 2 and

$${\Lambda }_{n,t}\left(\pi ,{\theta }^{\pi }\right)=\frac{1}{n}{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{\upsilon =1}^{{}_t}\frac{\pi \left(A_i^{\upsilon };S_i^{\upsilon }\right)}{{\widehat{\pi }}_n^{\upsilon -1}\left(A_i^{\upsilon };S_i^{\upsilon }\right)}}}\left\{U_i^{\upsilon }+\gamma V\left(\pi ,S_i^{\upsilon +1};{\theta }^{\pi }\right)-V\left(\pi ,S_i^{\upsilon };{\theta }^{\pi }\right)\right\}{\nabla }_{{\theta }^{\pi }}V\left(\pi ,S_i^{\upsilon };{\theta }^{\pi }\right)$$ (7)

with ${\widehat{\pi }}_n^0$ some initial randomized policy. We note that estimating equation (7) is similar to (3), except that ${\widehat{\pi }}_n^{\upsilon -1}$ replaces μ^υ as the data-generating policy. Given the estimator of the value of π at time t, ${\widehat{V}}_{n,\mathcal{R},t}\left(\pi \right)=\int V\left(\pi ,s;{\widehat{\theta }}_{n,t}^{\pi }\right)d\mathcal{R}\left(s\right)$, the estimated optimal policy at time t is ${\widehat{\pi }}_n^t=\text{arg}{\text{max}}_{\pi \in \Pi }{\widehat{V}}_{n,\mathcal{R},t}\left(\pi \right)$. In practice, we may choose to update the policy in batches rather than at every time point. An alternative way to encourage exploration through the action space is to choose ${\widehat{\pi }}_n^t=\text{arg}{\text{max}}_{\pi \in \Pi }\left\{{\widehat{V}}_{n,\mathcal{R},t}\left(\pi \right)+{\alpha }^t{\widehat{\psi }}^t\left(\pi \right)\right\}$ for some sequence α^t ≥ 0, where ${\widehat{\psi }}^t\left(\pi \right)$ is a measure of uncertainty in ${\widehat{V}}_{n,\mathcal{R},t}\left(\pi \right)$. An example of this is upper confidence bound sampling, or UCB (Lai and Robbins, 1985).

In some settings, when the data-generating process may vary across patients, it may be desirable to allow each patient to follow an individualized policy that is estimated using only that patient’s data. Suppose that n patients are followed for an initial T₁ time points after which the policy ${\widehat{\pi }}_n^1$ is estimated. Then, suppose that patient i follows ${\widehat{\pi }}_n^1$ until time T₂, when a policy ${\widehat{\pi }}_i^2$ is estimated using only the states and actions observed for patient i. This procedure is then carried out until time T_K for some fixed K with each patient following their own individual policy which is adapted to match the individual over time. We may also choose to adapt the randomness of the policy at each estimation. For example, we could select ϵ₁ > ϵ₂ > … > ϵ_K and, following estimation k, have patient i follow policy ${\widehat{\pi }}_i^k$ with probability 1 – ϵ_k and policy ${\widehat{\pi }}_n^1$ with probability ϵ_k. In this way, patients become more likely to follow their own individualized policy and less likely to follow the initial policy over time, reflecting increasing confidence in the individualized policy as more data become available. The same class of policies and model for the state value function can be used as in Section 2.

4. Theoretical results

In this section, we establish asymptotic properties of ${\widehat{\theta }}_n^{\pi }$ and ${\widehat{\pi }}_n$ for offline estimation. Because the proposed online estimation procedure involves performing offline estimation repeatedly in smaller batches, our theoretical results apply to each smaller batch as the number of observations increases. Developing more general theory for online estimation is an interesting topic for future research. Throughout, we assume assumptions 1–3 from Section 2.

Let ${\widehat{\theta }}_n^{\pi }=\text{arg}{\text{min}}_{{\theta }^{\pi }}{}_{\in \Theta }\left\{{\Lambda }_n{\left(\pi ,{\theta }^{\pi }\right)}^{⊺}\Omega {\Lambda }_n\left(\pi ,{\theta }^{\pi }\right)+{\lambda }_n{\left({\theta }^{\pi }\right)}^{⊺}{\theta }^{\pi }\right\}$. Thus, we consider the special case where the penalty function is the squared Euclidean norm of θ^π. We will assume that λ_n = o_P(n^−1/2). All of our results hold for any positive definite matrix, Ω. Assume the working model for the state value function introduced in Section 2, i.e., $V\left(\pi ,s_i^t;{\theta }^{\pi }\right)=\Phi {\left(s_i^t\right)}^{⊺}{\theta }^{\pi }$. For fixed π, denote the true ${\theta }^{\pi }by{\theta }_0^{\pi }$, i.e., $V\left(\pi ,s\right)=\Phi {\left(s\right)}^{⊺}{\theta }_0^{\pi }$. Let $\nu =\int \Phi \left(s\right)\text{d}\mathcal{R}\left(s\right)$ so that $V_{\mathcal{R}}\left(\pi \right)={\nu }^{⊺}{\theta }_0^{\pi }$. Define ${\widehat{V}}_{n,\widehat{\mathcal{R}}}\left(\pi \right)={\left\{{\mathbb{E}}_n\Phi \left(S\right)\right\}}^{⊺}{\widehat{\theta }}_n^{\pi }$, where ${\mathbb{E}}_n$ denotes the empirical measure of the observed data. Let $\Pi =\left\{{\pi }_{\beta }:\beta \in \mathcal{B}\right\}$ be a parametric class of policies and let ${\widehat{\pi }}_n={\pi }_{{\widehat{\beta }}_n}$ where ${\widehat{\beta }}_n=\text{arg}{\text{max}}_{\beta \in \mathcal{B}}{\widehat{V}}_{n,\widehat{\mathcal{R}}}\left({\pi }_{\beta }\right)$.

Our main results are summarized in Theorems 4.2 and 4.3 below. Because each patient trajectory is a stationary Markov chain, we need to use asymptotic theory based on stationary processes; consequently, some of the required technical conditions are more difficult to verify than those for i.i.d. data. Define the bracketing integral for a class of functions, $\mathcal{F}$, by $J_{[]}\left\{\delta ,\mathcal{F},L_r\left(P\right)\right\}={\int }_0^{\delta }\sqrt{\text{log}{N}_{[]}\left\{ϵ,\mathcal{F},L_r\left(P\right)\right\}}\text{d}ϵ,$ where the bracketing number for $\mathcal{F}$, $N_{[]}\left\{ϵ,\mathcal{F},L_r\left(P\right)\right\}$, is the number of L_r(P) ϵ-brackets needed such that each element of $\mathcal{F}$ is contained in at least one bracket (see Chapter 2 of Kosorok, 2008). For any stationary sequence of possibly dependent random variables, {X^t}_t≥1; let ${\mathcal{M}}_b^c$ be the σ-field generated by X^b, … ,X^c and define $\zeta \left(k\right)=\mathbb{E}\left[{\text{sup}}_{m\ge 1}\left\{\left|P\left(B|{\mathcal{M}}_1^m\right)-P\left(B\right)\right|:B\in {\mathcal{M}}_{m+k}^{\infty }\right\}\right]$. We say that the chain {X^t}_t≥1 is absolutely regular if ζ(k) → 0 as k → 0 (also called β-mixing in Chapter 11 of Kosorok, 2008). We make the following assumptions.

Assumption 4. There exists a 2 < ρ < ∞ such that

1. $\mathbb{E}{\left|U^t\right|}^{3\rho }<\infty ,\mathbb{E}{\Vert \Phi \left(S^t\right)\Vert }^{3\rho }<\infty$, and $\mathbb{E}{\Vert S^t\Vert }^{3\rho }<\infty$.

2. The sequence {(S^t, A^t)}_t≥1 is absolutely regular with ${\displaystyle {\sum }_{k=1}^{\infty }{k}^{2/\left(\rho -2\right)}}\zeta \left(k\right)<\infty$.

3. The bracketing integral of the class of policies, J_[] (∞,Π, L_3ρ(P)} < ∞.

Assumption 5. There exists some c₁ > 0 such that

$$\underset{\pi \in \Pi }{\text{inf}}{c}^{⊺}\mathbb{E}\left[\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}\left\{\Phi \left(S^t\right)\Phi {\left(S^t\right)}^{⊺}-{\gamma }^2\Phi \left(S^{t+1}\right)\Phi {\left(S^{t+1}\right)}^{⊺}\right\}\right]c\ge c_1{\Vert c\Vert }^2$$

for all $c\in {ℝ}^q$

Assumption 6. The map $\beta \mapsto V_{\mathcal{R}}\left({\pi }_{\beta }\right)$ has a unique and well separated maximum over β in the interior of $\mathcal{B}$; let β₀ denote the maximizer.

Assumption 7. The following condition holds: ${\text{sup}}_{\Vert {\beta }_1-{\beta }_2\Vert \le \delta }\mathbb{E}\Vert {\pi }_{{\beta }_1}\left(A;S\right)-{\pi }_{{\beta }_2}\left(A;S\right)\Vert \to 0$ as δ ↓0.

Remark 4.1. Assumption 4 requires certain finite moments and that the dependence between observations on the same patient vanishes as observations become further apart. In Lemma 8.2 in the appendix, we verify part 3 of assumption 4 and assumption 7 for the class of policies introduced in Section 2. However, note that the theory holds for any class of policies satisfying the given assumptions, not just the class considered here. Assumption 5 is needed to show the existence of a unique ${\theta }_0^{\pi }$ uniformly over Π and can be verified empirically by checking that certain data-dependent matrices are invertible. Assumption requires that the true optimal decision in each state is unique (see assumption A. 8 of Ertefaie, 2014) and is a standard assumption in M-estimation (see chapter 14 of Kosorok, 2008). Assumption 7 requires smoothness on the class of policies.

The main results of this section are stated below. Theorem 4.2 states that there exists a unique solution to $0=\mathbb{E}{\Lambda }_n\left(\pi ,{\theta }^{\pi }\right)$ uniformly over Π and that the estimator ${\widehat{\theta }}_n$ converges weakly to a mean zero Gaussian process in ${\ell }^{\infty }(\Pi )$.

Theorem 4.2. Under the given assumptions, the following hold.

1. For all π ∈ Π, there exists $a{\theta }_0^{\pi }\in {ℝ}^q$ such that $\mathbb{E}{\Lambda }_n\left(\pi ,{\theta }^{\pi }\right)$ has a zero at ${\theta }^{\pi }={\theta }_0^{\pi }$. Moreover, ${\text{sup}}_{\pi \in \Pi }\Vert {\theta }_0^{\pi }\Vert <\infty$ and ${\text{sup}}_{\Vert {\beta }_1-{\beta }_2\Vert \le \delta }\Vert {\theta }_0^{{\pi }_{{\beta }_1}}-{\theta }_0^{{\pi }_{{\beta }_2}}\Vert \to 0\text{as}\delta \downarrow 0$.

2. Let $\mathbb{G}\left(\pi \right)$ be a tight, mean zero Gaussian process indexed by Π with covariance $\mathbb{E}\left\{\mathbb{G}\left({\pi }_1\right)\mathbb{G}\left({\pi }_2\right)\right\}=w_1{\left({\pi }_1\right)}^{-1}{w}_0\left({\pi }_1,{\pi }_2\right)w_1{\left({\pi }_2\right)}^{-⊺}$ where

   $$w_0\left({\pi }_1,{\pi }_2\right)=\mathbb{E}\left[\frac{{\pi }_1\left(A^t;S^t\right){\pi }_2\left(A^t;S^t\right)}{{\mu }^t{\left(A^t;S^t\right)}^2}\left\{U^t+\gamma \Phi \left(S^{t+1}\right){\theta }_0^{{\pi }_1}-\Phi \left(S^t\right){\theta }_0^{{\pi }_1}\right\}\left\{U^t+\gamma \Phi \left(S^{t+1}\right){\theta }_0^{{\pi }_2}-\Phi \left(S^t\right){\theta }_0^{{\pi }_2}\right\}\Phi \left(S^t\right)\Phi {\left(S^t\right)}^{⊺}\right]$$

   and

   $$w_1\left(\pi \right)=\mathbb{E}\left[\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}\Phi \left(S^t\right)\left\{\Phi \left(S^t\right)-\gamma \Phi {\left(S^{t+1}\right)}^{⊺}\right\}\right].$$

   Then, $\sqrt{n}\left({\widehat{\theta }}_n^{\pi }-{\theta }_0^{\pi }\right)⇝\mathbb{G}\left(\pi \right)$ in ${\ell }^{\infty }(\Pi )$.

3. Let $\mathbb{G}\left(\pi \right)$ be as defined in part 2. Then, $\sqrt{n}\left\{{\widehat{V}}_{n,\widehat{\mathcal{R}}}\left(\pi \right)-V_{\mathcal{R}}\left(\pi \right)\right\}⇝{\nu }^{⊺}\mathbb{G}\left(\pi \right)$ in ${\ell }^{\infty }(\Pi )$.

Theorem 4.3 below gives us that the estimated optimal policy converges in probability to the true optimal policy over Π and that the estimated value of the estimated optimal policy converges to the true value of the estimated optimal policy.

Theorem 4.3. Under the given assumptions, the following hold.

1. Let ${\widehat{\beta }}_n=\text{arg}{\text{max}}_{\beta \in \mathcal{B}}{\widehat{V}}_{n,\widehat{\mathcal{R}}}\left({\pi }_{\beta }\right)$ and ${\beta }_0=\text{arg}{\text{max}}_{\beta \in \mathcal{B}}{V}_{\mathcal{R}}\left({\pi }_{\beta }\right)$. We have that $\Vert {\widehat{\beta }}_n-{\beta }_0\Vert \stackrel{P}{\to }0$.

2. Let ${\widehat{\beta }}_n$ and β₀ be defined as in part 1. Then, $\left|V_{\mathcal{R}}\left({\pi }_{{\widehat{\beta }}_n}\right)-V_{\mathcal{R}}\left({\pi }_{{\widehat{\beta }}_0}\right)\right|\stackrel{P}{\to }0$.

3. Let ${\sigma }_0^2={\nu }^{⊺}{w}_1{\left({\pi }_{{\beta }_0}\right)}^{-1}{w}_0\left({\pi }_{{\beta }_0},{\pi }_{{\beta }_0}\right)w_1\left({\pi }_{{\beta }_0}\right){}^{-⊺}\nu .$ Then, $\sqrt{n}\left\{{\widehat{V}}_{n,\widehat{\mathcal{R}}}\left({\pi }_{{\widehat{\beta }}_n}\right)-V_{\mathcal{R}}\left({\pi }_{{\widehat{\beta }}_n}\right)\right\}⇝N\left(0,{\sigma }_0^2\right)$.

4. A consistent estimator for ${\sigma }_0^2$ is

   $${\widehat{\sigma }}_n^2={\left\{\mathbb{E}\Phi \left(S^t\right)\right\}}^{⊺}{\widehat{w}}_1{\left({\pi }_{{\widehat{\beta }}_n}\right)}^{-1}{\widehat{w}}_0\left({\pi }_{{\widehat{\beta }}_n},{\pi }_{{\widehat{\beta }}_n}\right){\widehat{w}}_1{\left({\pi }_{{\widehat{\beta }}_n}\right)}^{-⊺}\left\{{\mathbb{E}}_n\Phi \left(S^t\right)\right\},$$

   where

   $${\widehat{w}}_0\left({\pi }_1,{\pi }_2\right)={\mathbb{E}}_n\left[\frac{{\pi }_1\left(A^t;S^t\right){\pi }_2\left(A^t;S^t\right)}{{\mu }^t{\left(A^t;S^t\right)}^2}\left\{U^t+\gamma \Phi \left(S^{t+1}\right){\widehat{\theta }}_n^{{\pi }_1}-\Phi \left(S^t\right){\widehat{\theta }}_0^{{\pi }_1}\right\}\left\{U^t+\gamma \Phi \left(S^{t+1}\right){\widehat{\theta }}_n^{{\pi }_2}-\Phi \left(S^t\right){\widehat{\theta }}_n^{{\pi }_2}\right\}\Phi \left(S^t\right)\Phi {\left(S^t\right)}^{⊺}\right]$$

   and

   $${\widehat{w}}_1\left(\pi \right)={\mathbb{E}}_n\left[\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}\Phi \left(S^t\right){\left\{\Phi \left(S^t\right)-\gamma \Phi \left(S^{t+1}\right)\right\}}^{⊺}\right].$$

Proofs of the above results are in the Appendix along with a result on bracketing entropy that is needed for the proof of Theorem 4.2 and a proof that the class of policies introduced above satisfies the necessary bracketing integral assumption.

5. Simulation experiments

In this section, we examine the performance of V-learning on simulated data. Section 5.1 contains results for offline estimation and Section 5.2 contains results for online estimation. All simulation results are averaged across 50 replications.

5.1. Offline simulations

Our implementation of V-learning follows the setup in Section 2. Maximizing ${\widehat{V}}_{n,\mathcal{R}}\left(\pi \right)$ is done using a combination of simulated annealing and the BFGS algorithm as implemented in the optim function in R software (R Core Team, 2016). We note that ${\widehat{V}}_{n,\mathcal{R}}\left(\pi \right)$ is differentiable in π, thereby avoiding some of the computational complexity of GGQ. However, the objective is not necessarily convex. In order to avoid local maxima, simulated annealing with 1000 function evaluations is used to find a neighborhood of the maximum; this solution is then used as the starting value for the BFGS algorithm.

We use the class of policies introduced in Section 2. Although we maximize the value over a class of randomized policies, the true optimal policy is deterministic. To prevent the coefficients of ${\widehat{\beta }}_n$ from diverging to infinity, we add an L2 penalty when maximizing over β. To prevent overfitting, we use an L2 penalty when computing ${\widehat{\theta }}_n^{\pi }$, i.e., $\mathcal{P}\left({\theta }^{\pi }\right)={\left({\theta }^{\pi }\right)}^{⊺}{\theta }^{\pi }$. We let Ω be the identity matrix. Simulation results with alternate choices of $\mathcal{P}$ and Ω are given in the appendix. Tuning parameters can be used to control the amount of randomness in the estimated policy. For example, increasing the penalty when computing ${\widehat{\beta }}_n$ is one way to encourage exploration through the action space because β = 0 defines a policy where each action is selected with equal probability.

We consider three different models for the state-value function: (i) linear; (ii) second degree polynomial; and (iii) Gaussian radial basis functions (RBF). The Gaussian RBF is $\varphi \left(x;\kappa ,{\tau }^2\right)=\text{exp}\left\{-{\left(x-\kappa \right)}^2/2{\tau }^2\right\}$. We use τ = 0.25 and κ = 0, 0.25, 0.5, 0.75, 1 to create a basis of functions and apply this basis to the state variables after scaling them to be between 0 and 1. Each model also implicitly contains an intercept.

We begin with the following simple generative model. Let the two-dimensional state vector be $S_i^t={\left(S_{i,1}^t,S_{i,2}^t\right)}^{⊺}$, i = 1, … , n, t = 1, … , T. We initiate the state variables as independent standard normal random variables and let them evolve according to $S_{i,1}^t=\left(3/4\right)\left(2A_i^{t-1}-1\right)S_{i,1}^{t-1}+\left(1/4\right)S_{i,1}^{t-1}{S}_{i,2}^{t-1}+{ϵ}_1^t$ and $S_{i,2}^t=\left(3/4\right)\left(1-2A_i^{t-1}\right)S_{i,2}^{t-1}+\left(1/4\right)S_{i,1}^{t-1}{S}_{i,2}^{t-1}+{ϵ}_2^t$, where $A_i^t$ takes values in {0, 1} and ${ϵ}_1^t$ and ${ϵ}_2^t$ are independent N (0, 1/4) random variables. Define the utility function by $U_i^t=u\left(S_i^{t+1},A_i^t,S_i^t\right)=2S_{i,1}^{t+1}+S_{i,2}^{t+1}-\left(1/4\right)\left(2A_i^t-1\right)$. At each time t, we must make a decision to treat or not with the goal of maximizing the components of S while treating as few times as possible. Treatment has a positive effect on S₁ and a negative effect on S₂. We generate $A_i^t$ from a Bernoulli distribution with mean 1/2. In estimation, we assume that the generating model for treatment is known, as would be the case in a micro-randomized trial.

We generate samples of n patients with T time points per patient from the given generative model after an initial burn-in period of 50 time points. The burn-in period ensures that our simulated data are sampled from an approximately stationary distribution. We estimate policies using V-learning with three different types of basis functions and GGQ. After estimating optimal policies, we simulate 100 patients following each estimated policy for 100 time points and take the mean utility under each policy as an estimate of the value of that policy. Estimated values are found in Table 1 with Monte Carlo standard errors along with the observed value. Recall that larger values are better. The policies estimated using V-learning produce better outcomes than the observational policy and the policy estimated using GGQ. V-learning produces the best outcomes using Gaussian basis functions. In the Appendix, we give results for the same simulation settings given in Table 1 for alternate choices of $\mathcal{P}$ and Ω. Table 8 presents results with the LASSO penalty when Ω is the identity matrix, Table 9 presents results with the L2 penalty when $\Omega ={\left\{{\mathbb{E}}_n{S}^t{\left(S^t\right)}^{⊺}\right\}}^{-1}$, and Table 10 presents results with the LASSO penalty when $\Omega ={\left\{{\mathbb{E}}_n{S}^t{\left(S^t\right)}^{⊺}\right\}}^{-1}$.

Table 1:

Monte Carlo value estimates for offline simulations with γ = 0.9.

n	T	Linear VL	Polynomial VL	Gaussian VL	GGQ	Observed
25	24	0.118 (0.0892)	0.091 (0.0825)	0.110 (0.0979)	0.014 (0.0311)	−0.005
	36	0.108 (0.0914)	0.115 (0.0911)	0.112 (0.0919)	0.029 (0.0280)	−0.004
	48	0.106 (0.0705)	0.071 (0.0974)	0.103 (0.0757)	0.031 (0.0350)	0.000
50	24	0.124 (0.0813)	0.109 (0.1045)	0.118 (0.0879)	0.016 (0.0355)	−0.005
	36	0.126 (0.0818)	0.134 (0.0878)	0.136 (0.0704)	0.027 (0.0276)	0.003
	48	0.101 (0.0732)	0.109 (0.0767)	0.115 (0.0763)	0.020 (0.0245)	0.000
100	24	0.117 (0.0895)	0.135 (0.0973)	0.140 (0.0866)	0.019 (0.0257)	0.011
	36	0.113 (0.0853)	0.105 (0.1033)	0.139 (0.0828)	0.021 (0.0312)	0.012
	48	0.111 (0.0762)	0.143 (0.0853)	0.114 (0.0699)	0.031 (0.0306)	−0.001

Next, we simulate cohorts of patients with type 1 diabetes to mimic the mHealth study of Maahs et al. (2012). Maahs et al. (2012) followed a small sample of youths with type 1 diabetes and recorded data at a fine granularity using mobile devices. Blood glucose levels were tracked in real time using continuous glucose monitoring, physical activity was measured continuously using accelerometers, and insulin injections were logged by an insulin pump. Dietary data were recorded by 24-hour recall over phone interviews.

In our simulation study, we divide each day of follow-up into 60 minute intervals. Thus, for one day of follow-up, we observe T = 24 time points per simulated patient and a treatment decision is made every hour. Our hypothetical mHealth study is designed to estimate an optimal dynamic treatment regime for the timing of insulin injections based on patient blood glucose, physical activity, and dietary intake with the goal of controlling future blood glucose as close as possible to the optimal range. To this end, we define the utility at time t as a weighted sum of hypo- and hyperglycemic episodes in the 60 minutes preceding and following time t. Weights are −3 when glucose ≤ 70 (hypoglycemic), −2 when glucose ≤ 150 (hyperglycemic), −1 when 70 < glucose ≤ 80 or 120 < glucose ≤ 150 (borderline hypo- and hyperglycemic), and 0 when 80 < glucose ≤ 120 (normal glycemia). Utility at each time point ranges from −6 to 0 with larger utilities (closer to 0) being more preferable. For example, a patient who presents with an average blood glucose of 155 mg/dL over time interval t–1, takes an action to correct their hyperglycemia, and presents with an average blood glucose of 145 mg/dL over time interval t would receive a utility of U^t = −3. Weights were chosen to reflect the relative clinical consequences of high and low blood glucose. For example, acute hypoglycemia, characterized by blood glucose levels below 70 mg/dL, is an emergency situation that can result in coma or death.

Simulated data are generated as follows. At each time point, patients are randomly chosen to receive an insulin injection with probability 0.3, consume food with probability 0.2, partake in mild physical activity with probability 0.4, and partake in moderate physical activity with probability 0.2. Grams of food intake and counts of physical activity are generated from normal distributions with parameters estimated from the data of Maahs et al. (2012). Initial blood glucose level for each patient is drawn from a normal distribution with mean 100 and standard deviation 25. Define the covariates for patient i collected at time t by ${\left({\text{Gl}}_i^t,{\text{Di}}_i^t,{\text{Ex}}_i^t\right)}^{⊺}$, where ${\text{Gl}}_i^t$ is average blood glucose level, ${\text{Di}}_i^t$ is total dietary intake, and ${\text{Ex}}_i^t$ is total counts of physical activity as would be measured by an accelerometer. Glucose levels evolve according to

$${\text{Gl}}^t=\mu \left(1-{\alpha }_1\right)+{\alpha }_1{\text{Gl}}^{t-1}+{\alpha }_2{\text{Di}}^{t-1}+{\alpha }_3{\text{Di}}^{t-2}+{\alpha }_4{\text{Ex}}^{t-1}+{\alpha }_{5}E{x}^{t-2}+{\alpha }_6{\text{In}}^{t-1}+{\alpha }_7{\text{In}}^{t-2}+e,$$ (8)

where In^t is an indicator of an insulin injection received at time t and e ~ N(0, σ²). We use the parameter vector α = (α₁, … , α₇)^⊺ = (0.9, 0.1, 0.1, −0.01, −0.01, −2, −4)^⊺, μ = 100, and σ = 5.5 based on a linear model fit to the data of Maahs et al. (2012). The known lag-time in the effect of insulin is reflected by α₆ = −2 and α₇ = −4. Selecting α₁ < 1 ensures the existence of a stationary distribution.

We define the state vector for patient i at time t to contain average blood glucose, total dietary intake, and total physical activity measured over previous time intervals; we include blood glucose and physical activity for the previous two time intervals and dietary intake for the previous four time intervals. Let n denote number of patients and T denote number of time points per patient. Our choices for n and T are based on what is feasible for an mHealth outpatient study (dietary data was collected on two days by Maahs et al., 2012). For each replication, the optimal treatment regime is estimated with V-learning using three different types of basis functions and GGQ. The generative model for insulin treatment is not assumed to be known and we estimate it using logistic regression. We record mean outcomes in an independent sample of 100 patients followed for 100 time points with treatments generated according to each estimated optimal regime. Simulation results (estimated values under each regime and Monte Carlo standard errors along with observed values) are found in Table 2. Again, V-learning with Gaussian basis functions performs the best out of all methods, generally producing large values and small standard errors. V-learning with the linear model underperforms and GGQ underperforms somewhat more so.

Table 2:

Monte Carlo value estimates for simulated T1D cohorts with γ = 0.9.

n	T	Linear VL	Polynomial VL	Gaussian VL	GGQ	Observed
25	24	−2.716 (1.2015)	−2.335 (0.9818)	−2.018 (1.2011)	−3.870 (0.9225)	−2.316
	36	−2.700 (1.2395)	−2.077 (1.0481)	−1.760 (0.8468)	−3.644 (0.8745)	−2.261
	48	−2.496 (1.1986)	−2.236 (1.1978)	−1.751 (0.9887)	−2.405 (1.1025)	−2.365
50	24	−2.545 (1.1865)	−2.069 (1.0395)	−1.605 (0.8064)	−3.368 (1.0186)	−2.263
	36	−2.644 (1.1719)	−2.004 (0.9074)	−1.778 (0.8496)	−3.099 (0.9722)	−2.336
	48	−2.469 (1.1635)	−2.073 (0.9870)	−2.102 (1.2078)	−2.528 (0.9571)	−2.308
100	24	−2.350 (1.1171)	−2.128 (1.0520)	−1.612 (0.7203)	−3.272 (0.8636)	−2.299
	36	−2.547 (1.1852)	−2.116 (0.8518)	−1.672 (0.8643)	−3.232 (0.7951)	−2.321
	48	−2.401 (1.0643)	−2.204 (1.0400)	−1.494 (0.5413)	−2.820 (0.8442)	−2.351

5.2. Online simulations

In practice, it may be useful for patients to follow a dynamic treatment regime that is updated as new data are collected. Here we consider a hypothetical study wherein n patients are followed for an initial period of T′ time points, an optimal policy is estimated, and patients are followed for an additional T – T′ time points with the estimated optimal policy being continuously updated. At each time point, t ≥ T′, actions are taken according to the most recently estimated policy. Recall that V-learning produces a randomized decision rule from which to sample actions at each time point. When selecting an action based on a GGQ policy, we incorporate an ϵ-greedy strategy by selecting the action recommended by the estimated policy with probability 1 – ϵ and otherwise randomly selecting one of the other actions. At the tth estimation, we use ϵ = 0.5^t, allowing ϵ to decrease over time to reflect increasing confidence in the estimated policy. A burn-in period of 50 time points is discarded to ensure that we are sampling from a stationary distribution. We estimate the first policy after 12 time points and a new policy is estimated every 6 time points thereafter. After T time points, we estimate the value as the average utility over all patients and all time points after the initial period.

Table 3 presents mean outcomes under policies estimated online using data generated according to the simple two covariate generative model introduced at the beginning of Section 5.1. There is some variability across n and T regarding which type of basis function is best, but V-learning with a polynomial basis generally produces the best outcomes. GGQ performs well in large samples.

Table 3:

Value estimates for online simulations with γ = 0.9.

n	T	Linear VL	Polynomial VL	Gaussian VL	GGQ
25	24	0.0053	0.0149	−0.0100	−0.0081
	36	0.0525	0.0665	0.0310	0.0160
	48	0.0649	0.0722	0.0416	0.0493
50	24	0.0164	0.0117	0.0037	0.0058
	36	0.0926	0.0791	0.0666	0.0227
	48	0.1014	0.0894	0.0512	0.0434
100	24	0.0036	−0.0157	0.0200	0.0239
	36	0.0766	0.0626	0.0907	0.0540
	48	0.0728	0.0781	0.0608	0.0818

Next, we study the performance of online V-learning in simulated mHealth studies of type 1 diabetes by following the generative model described in (8). Mean outcomes are found in Table 4. Gaussian V-learning performs the best out of all methods. Across all variants of V-learning, outcomes improve with increased follow-up time.

Table 4:

Value estimates for online estimation of simulated T1D cohorts with γ = 0.9

n	T	Linear VL	Polynomial VL	Gaussian VL	GGQ
25	24	−2.3887	−1.9713	−1.8860	−3.2027
	36	−2.3784	−2.1535	−1.7857	−3.5127
	48	−2.2190	−2.0679	−1.6999	−3.2280
50	24	−2.3405	−2.2313	−1.7761	−2.8976
	36	−2.2829	−2.0922	−1.6016	−3.1589
	48	−2.1587	−1.9669	−1.5948	−2.8729
100	24	−2.3229	−2.2295	−1.9138	−3.0865
	36	−2.2927	−2.1608	−1.9030	−3.3483
	48	−2.2096	−2.0454	−1.8252	−2.9428

Finally, we consider online simulations using individualized policies as outlined at the end of Section 3. Consider the simple two covariate generative model introduced above but let state variables evolve according to $S_{i,1}^t={\mu }_i\left(2A_i^{t-1}-1\right)S_{i,1}^{t-1}+\left(1/4\right)S_{i,1}^{t-1}{S}_{i,2}^{t-1}+{ϵ}_1^t$ and $S_{i,2}^t={\mu }_i\left(1-2A_i^{t-1}\right)S_{i,2}^{t-1}+\left(1/4\right)S_{i,1}^{t-1}{S}_{i,2}^{t-1}+{ϵ}_2^t$ where μ_i is a subject-specific term drawn uniformly between 0.4 and 0.9. Including μ_i ensures that the optimal policy differs across patients. Table 5 contains mean outcomes for online simulation where a universal policy is estimated using data from all patients and where individualized policies are estimated using only a single patient’s data. Because data are generated in a such a way that the optimal policy varies across patients, individualized policies achieve better outcomes than universal policies.

Table 5:

Value estimates for online V-learning simulations with universal and patient-specific policies when γ = 0.9.

n	T	Universal policy	Patient-specific policy
25	24	0.0282	0.1813
	36	0.1025	0.1700
	48	0.0977	0.1944
50	24	0.0164	0.2771
	36	0.0768	0.2617
	48	0.0752	0.3038
100	24	0.0160	0.4230
	36	0.0960	0.2970
	48	0.1140	0.3197

6. Case study: Type 1 diabetes

Machine learning is currently under consideration in type 1 diabetes through studies to build and test a “closed loop” system that joins continuous blood glucose monitoring and subcutaneous insulin infusion through an underlying algorithm. Known as the artificial pancreas, this technology has been shown to be safe in preliminary studies and is making headway from small hospital-based safety studies to large-scale outpatient effectiveness studies (Ly et al., 2014, 2015). Despite the success of the artificial pancreas, the rate of uptake may be limited and widespread use may not occur for many years (Kowalski, 2015). The proposed method may be useful for implementing mHealth interventions for use alongside the artificial pancreas or before it is widely available.

Studies have shown that data on food intake and physical activity to inform optimal decision making can be collected in an inpatient setting (see, e.g., Cobry et al., 2010; Wolever and Mullan, 2011). However, Maahs et al. (2012) demonstrated that rich data on the effect of food intake and physical activity can be collected in an outpatient setting using mobile technology. Here, we apply the proposed methodology to the observational data collected by Maahs et al. (2012).

The full data consist of N = 31 patients with type 1 diabetes, aged 12–18. Glucose levels were monitored using continuous glucose monitoring and physical activity tracked using accelerometers for five days. Dietary data were self-reported by the patient in telephone-based interviews for two days. Patients were treated using either an insulin pump or multiple daily insulin injections. We use data on a subset of n = 14 patients treated with an insulin pump for whom full follow-up is available on days when dietary information was recorded. This represents 28 patient-days of data, with which we use V-learning to estimate an optimal treatment policy. An advantage of mHealth is the ability to collect data passively, limiting the amount of missing data. There is no intermittent missingness in this data set.

The setup closely follows the simulation experiments in Section 5.1. Patient state at each time, t, is taken to be average glucose level and total counts of physical activity over the two previous 60 minute intervals and total food intake in grams over the four previous 60 minute intervals. The goal is to learn a policy to determine when to administer insulin injections based on prior blood glucose, dietary intake, and physical activity. The utility at time t is a weighted sum of glycemic events over the 60 minutes preceding and following time t with weights defined in Section 5.1. A treatment regime with large value will minimize the number of hypo- and hyperglycemic episodes weighted to reflect the clinical importance of each. We note that because V(π, s; θ^π) is linear in θ^π, we can evaluate ${\widehat{V}}_{n,\widehat{\mathcal{R}}}\left(\pi \right)$ with only the mean of Φ(S) under $\mathcal{R}$. These were estimated from the data. Because we cannot simulate data following a given policy to estimate its value, we report the parametric value estimate ${\widehat{V}}_{n,\widehat{\mathcal{R}}}\left({\widehat{\pi }}_n\right)$. Interpreting the parametric value estimate is difficult because of the effect the discount factor has on estimated value. We cannot compare parametric value estimates to mean outcomes observed in the data. Instead, we use ${\mathbb{E}}_n{\displaystyle {\sum }_{t\ge 0}{\gamma }^t{U}^t}$ as an estimate of value under the observational policy.

We estimate optimal treatment strategies for two different action spaces. In the first, the only decision made at each time is whether or not to administer an insulin injection, i.e., the action space contains a single binary action. In the second, the action space contains all possible combinations of insulin injection, physical activity, and food intake. This corresponds to a hypothetical mHealth intervention where insulin injections are administered via an insulin pump and suggestions for physical activity and food intake are administered via a mobile app.

Table 6 contains parametric value estimates for policies estimated using V-learning for the two action spaces outlined above with different basis functions and discount factors. These results indicate that improvements in glycemic control can come from personalized and dynamic treatment strategies that account for food intake and physical activity. Improvement results from a dynamic insulin regimen (binary action space), and, in most cases, further improvement results from a comprehensive mHealth intervention including suggestions for diet and exercise delivered via mobile app in addition to insulin therapy (multiple action space). When considering multiple actions, the policy estimated using a polynomial basis and γ = 0.7 achieves a 64% increase in value and the policy estimated using a Gaussian basis and γ = 0.8 achieves a 68% increase in value over the observational policy. Although the small sample size is a weakness of this study, these results represent a significant improvement in value despite the sample size.

Table 6:

Parametric value estimates for V-learning applied to type 1 diabetes data.

Action space	Basis	γ = 0.7	γ = 0.8	γ = 0.9
Binary	Linear	−6.20	−9.35	−15.99
	Polynomial	−3.91	−9.03	−17.50
	Gaussian	−3.44	−13.09	−25.52
Multiple	Linear	−6.47	−9.92	−0.49
	Polynomial	−2.44	−6.80	−14.48
	Gaussian	−8.45	−3.58	−21.18
Observational policy		−6.77	−11.28	−21.79

Finally, we use an example hyperglycemic patient to illustrate how an estimated policy would be applied in practice. One patient in the data presented at a specific time with an average blood glucose of 229 m g/dL over the previous hour and an average blood glucose of 283 m g/dL over the hour before that. The policy estimated with γ = 0.7 and a polynomial basis recommends each action according to the probabilities in Table 7. Because this patient presented with blood glucose levels that are higher than the optimal range, the policy recommends actions that would lower the patient’s blood glucose levels, assigning a probability of 0.79 to insulin and a probability of 0.21 to insulin combined with activity.

Table 7:

Probabilities for each action as recommended by estimated policy for one example patient.

Action	Probability
No action	< 0.0001
Physical activity	< 0.0001
Food intake	< 0.0001
Food and activity	< 0.0001
Insulin	0.7856
Insulin and activity	0.2143
Insulin and food	0.0002
Insulin, food, and activity	< 0.0001

7. Conclusion

The emergence of mHealth has provided great potential for the estimation and implementation of dynamic treatment regimes. Mobile technologies can be used both in the collection of rich longitudinal data to inform decision making and the delivery of deeply tailored interventions. The proposed method, V-learning, addresses a number of challenges associated with estimating dynamic treatment regimes in mHealth applications. V-learning directly estimates a policy which maximizes the value over a class of policies and requires minimal assumptions on the data-generating process. Furthermore. V-learning permits estimation of a randomized decision rule which can be used in place of existing strategies (e.g., ϵ-greedy) to encourage exploration in online estimation. A randomized decision rule can also provide patients with multiple treatment options. Estimation of an optimal policy for different populations can be handled through the use of different reference distributions.

V-learning and mobile technologies have the potential to improve patient outcomes in a variety of clinical areas. We have demonstrated, for example, that the proposed method can be used to estimate treatment regimes to reduce the number of hypoand hyperglycemic episodes in patients with type 1 diabetes. The proposed method could also be useful for other mHealth applications as well as applications outside of mHealth. For example, V-learning could be used to estimate dynamic treatment regimes for chronic illnesses using electronic health records data. Future research in this area may include increasing flexibility through use of a semiparametric model for the state-value function. Alternatively, nonlinear models for the state-value function may be informed by underlying theory or mathematical models of the system of interest. Data-driven selection of tuning parameters for the proposed method may help to improve performance. Developing theory for alternative penalty functions, such as the LASSO penalty, is another important step. Accounting for patient availability and feasibility of a sequence of treatments can be done by setting constraints on the class of policies. This will ensure that the resulting mHealth intervention is able to be implemented and that the recommended decisions are consistent with domain knowledge.

It would also be worthwhile to generalize our asymptotic results to permit nonstationarity. While we believe that stationarity is generally a reasonable assumption for moderate stretches of time—including when we do online estimation using moderately large batches of observations wherein both the patient dynamics and treatment policy remain approximately constant over each batch—stationarity would not in general hold when either the patient dynamics or treatment policy change more rapidly. For example, online estimation with treatment policy changes after each observation could induce nonstationarity. We conjecture that our asymptotic results will continue to hold in this setting, as our simulation studies in Section 5.2 seem to indicate.

Appendix

Proofs

Proof of Lemma 2.1. Let π be an arbitrary policy and 7 γ ∈ (0, 1) a fixed constant. Suppose we observe a state S^t = s^t at time t and let ${\overline{a}}^{t-1}=\left(a^1,\dots ,a^{t-1}\right)$ be the sequence of actions resulting in S^t = s^t, i.e., $S^{\ast t}\left({\overline{a}}^{t-1}\right)=s^t$. Let ${\overline{a}}^{k+1}=\left(a^t,\dots ,a^{t+k}\right)\in {\mathcal{A}}^{k+1}$ be a potential sequence of actions taken from time t to time t + k. We have that

$$\begin{array}{lll}V\left(\pi ,{\mathit{s}}^t\right)\hfill & =\hfill & {\displaystyle \sum }_{k\ge 0}{\gamma }^k\mathbb{E}\left\{U^{\text{*}\left(t+k\right)}\left(\pi \right)\right|S^t=s^t\}\hfill \\ \hfill & =\hfill & {\displaystyle \sum }_{k\ge 0}{\gamma }^k\mathbb{E}\left({\displaystyle \sum }_{{\overline{a}}^{t+k}\in {\mathcal{A}}^{t+k}}{U}^{\text{*}\left(t+k\right)}\left({\overline{a}}^{t+k}\right){\displaystyle \prod }_{v=t}^{t+k}1\left[{\xi }_{\pi }^v\left\{{\mathit{S}}^{\text{*}v}\left({\overline{a}}^{v-1}\right)\right\}=a^v\right]\right|S^t=s^t)\hfill \\ \hfill & =\hfill & {\displaystyle \sum }_{k\ge 0}{\gamma }^k{\displaystyle \sum }_{{\overline{a}}^{k+1}\in {\mathcal{A}}^{k+1}}{U}^{\text{*}\left(t+k\right)}\left({\overline{a}}^{t-1},a^{-k+1}\right)\{{\displaystyle \prod }_{v=1}^{t+k}\mathbb{E}(1\left[{\xi }_{\pi }^v\left\{S^{\text{*}v}\left({\overline{a}}^{v-1}\right)\right\}=a^v\right]|S^t=s^t)\}\hfill \\ \hfill & =\hfill & {\displaystyle \sum }_{k\ge 0}{\gamma }^k{\displaystyle \sum }_{{\overline{a}}^{k+1}\in {\mathcal{A}}^{k+1}}{U}^{\text{*}\left(t+k\right)}\left({\overline{a}}^{t-1},a^{-k+1}\right){\displaystyle \prod }_{v=t}^{t+k}\pi \left\{a^v;S^{\text{*}v}\left({\overline{a}}^{v-1}\right)\right\}{\displaystyle \prod }_{v=t}^{t+k}\frac{{\mu }^v\left\{a^v;S^{\text{*}v}\left({\overline{a}}^{v-1}\right)\right\}}{{\mu }^v\left\{a^v;S^{\text{*}v}\left({\overline{a}}^{v-1}\right)\right\}}\hfill \\ \hfill & =\hfill & {\displaystyle \sum }_{k\ge 0}{\gamma }^k\mathbb{E}\left[U^{t+k}\left\{{\displaystyle \prod }_{v=0}^k\frac{\pi \left(a^{t+v};s^{t+v}\right)}{{\mu }^{t+v}\left(a^{t+v};s^{t+v}\right)}\right\}\right|S^t=s^t],\hfill \end{array}$$

where we let π(a^t; s^t) = 0 for all a^t and s^t whenever t > T*(π). The last equality uses the consistency and strong ignorability assumptions.

Proof of Theorem 4.2. Proof of part 1: We first note that ${\theta }_0^{\pi }$ must solve

$$0=\mathbb{E}\left(\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}\left[U^t+{\left\{\gamma \Phi \left(S^{t+1}\right)-\Phi \left(S^t\right)\right\}}^{\text{T}}{\theta }^{\pi }\right]\Phi \left(S^t\right)\right),$$

or

$$\mathbb{E}\left[\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}{\left\{\Phi \left(S^t\right)-\gamma \Phi \left(S^{t+1}\right)\right\}}^{\text{T}}\right]{\theta }^{\pi }=\mathbb{E}\left\{\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}{U}^t\Phi \left(S^t\right)\right\},$$

which is equivalent to $w_1\left(\pi \right){\theta }^{\pi }=w_2\left(\pi \right)$ where $w_2\left(\pi \right)=\mathbb{E}\left\{\pi \left(A^t;S^t\right){\mu }^t{\left(A^t;S^t\right)}^{-1}{U}^t\Phi \left(S^t\right)\right\}$. We have that

$$\Vert \mathbb{E}\left\{\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}{U}^t\Phi \left(S^t\right)\right\}\Vert \le c_0^{-1}{\left(\mathbb{E}{\left|U^t\right|}^2\right)}^{1/2}{\left(\mathbb{E}\Vert \Phi \left(S^t\right){\Vert }^2\right)}^{1/2}<\infty ,$$

by assumption 3, part 1 of assumption 4 and the Cauchy-Schwarz inequality. Let $c\in {ℝ}^q$ be arbitrary and note that

$$\mathbb{E}\left\{\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}{c}^{⊺}\Phi \left(S^t\right)\Phi {\left(S^{t+1}\right)}^{⊺}c\right\}\le {\left[\mathbb{E}\left\{\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}{c}^{⊺}\Phi {\left(S^t\right)}^{\otimes 2}c\right\}\cdot \mathbb{E}\left\{\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}{c}^{⊺}\Phi {\left(S^{t+1}\right)}^{\otimes 2}c\right\}\right]}^{1/2},$$

by the Cauchy-Schwarz inequality, where u^⊗2 = uu^⊺. This implies that

$$\begin{array}{lll}{c}^{⊺}{w}_1\left(\pi \right)c\hfill & \ge \hfill & \mathbb{E}\left\{\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}{c}^{⊺}\Phi {\left(S^t\right)}^{\otimes 2}c\right\}-\mathbb{E}{\left\{\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}{c}^{⊺}\Phi {\left(S^t\right)}^{\otimes 2}c\right\}}^{1/2}\mathbb{E}{\left\{{\gamma }^2\frac{\pi \left(A^t;S^t\right)}{{\mu }^t\left(A^t;S^t\right)}{c}^{⊺}\Phi {\left(S^{t+1}\right)}^{\otimes 2}c\right\}}^{1/2}\hfill \\ \hfill & =\hfill & A-A^{1/2}{B}^{1/2}\hfill \\ \hfill & =\hfill & A^{1/2}\left(A^{1/2}-B^{1/2}\right)\hfill \\ \hfill & =\hfill & \frac{A^{1/2}\left(A-B\right)}{A^{1/2}+B^{1/2}},\hfill \end{array}$$

where we simplify notation by defining $A=\mathbb{E}\left\{\pi \left(A^t;S^t\right){\mu }^t{\left(A^t;S^t\right)}^{-1}{c}^{⊺}\Phi {\left(S^t\right)}^{\otimes 2}c\right\}$ and $B=\mathbb{E}\left\{{\gamma }^2\pi \left(A^t;S^t\right){\mu }^t{\left(A^t;S^t\right)}^{-1}{c}^{⊺}\Phi {\left(S^{t+1}\right)}^{\otimes 2}c\right\}$. We have that

$$\begin{array}{lll}{A}^{1/2}+B^{1/2}\hfill & \le \hfill & c_0^{-1/2}\Vert c\Vert {\left\{\mathbb{E}{\Vert \Phi \left(S^t\right)\Vert }^2\right\}}^{1/2}+c_0^{-1/2}\Vert c\Vert {\left\{\mathbb{E}{\Vert \Phi \left(S^{t+1}\right)\Vert }^2\right\}}^{1/2}\hfill \\ \hfill & =\hfill & 2c_0^{-1/2}\Vert c\Vert {\left\{\mathbb{E}{\Vert \Phi \left(S^t\right)\Vert }^2\right\}}^{1/2}\hfill \\ \hfill & <\hfill & \infty ,\hfill \end{array}$$

by the Cauchy-Schwarz inequality, the fact that $\mathbb{E}{\Vert \Phi \left(S^t\right)\Vert }^2=\mathbb{E}{\Vert \Phi \left(S^{t+1}\right)\Vert }^2$ by time-homogeneity, and part 1 of assumption 4. Also, A ≥ A − B and A − B ≥ c₁∥c∥² by assumption 5. Thus,

$$\begin{array}{lll}A-A^{1/2}{B}^{1/2}\hfill & \ge \hfill & \frac{c_1^{3/2}{\Vert c\Vert }^3}{2c_0^{-1/2}\Vert c\Vert {\left\{\mathbb{E}{\Vert \Phi \left(S^t\right)\Vert }^2\right\}}^{1/2}}\hfill \\ \hfill & =\hfill & \frac{c_0^{1/2}{c}_1^{3/2}{\Vert c\Vert }^2}{2{\left\{\mathbb{E}{\Vert \Phi \left(S^t\right)\Vert }^2\right\}}^{1/2}},\hfill \end{array}$$

which finally implies that w₁(π) is invertible and thus ${\theta }_0^{\pi }=w_1{\left(\pi \right)}^{-1}{w}_2\left(\pi \right)$ is well-defined uniformly over π ∈ Π. Using the fact that c^⊺w₁(π)c ≥ k₀∥c∥² for a constant k₀ > 0, we can show that $\Vert w_1{\left(\pi \right)}^{-1}\Vert \le k_1^{-1}$ for some constant k₁ > 0, where ∥ · ∥ is the usual matrix norm when applied to a matrix. Therefore, $\Vert {\theta }_0^{\pi }\Vert \le k_1^{-1}\Vert w_2\left(\pi \right)\Vert \le c_0^{-1}{k}_1^{-1}{\left\{\mathbb{E}{\left(U^t\right)}^2\right\}}^{1/2}{\left\{\mathbb{E}{\Vert \Phi \left(S^t\right)\Vert }^2\right\}}^{1/2}<\infty$. Finally, it follows from assumptions 5 and 7 that ${\text{sup}}_{\Vert {\beta }_1-{\beta }_2\Vert \le \delta }\Vert {\theta }^{{\pi }_{{\beta }_1}}-{\theta }^{{\pi }_{{\beta }_2}}\Vert \to 0\text{as}\delta \downarrow 0$.

Proof of part 2: Define

$$\mathcal{G}=\left\{\Phi \left(s^t\right)\Phi {\left(s^t\right)}^{⊺}/{\mu }^t\left(a^t;s^t\right),\gamma \Phi \left(s^t\right)\Phi {\left(s^{t+1}\right)}^{⊺}/{\mu }^t\left(a^t;s^t\right),u^t\Phi \left(s^t\right)/{\mu }^t\left(a^t;s^t\right)\right\}.$$

Let G be an envelope for $\mathcal{G}$, for example $G\left(s^{t+1},a^t,s^t\right)={\text{max}}_{g\in \mathcal{G}}g\left(s^{t+1},a^t,s^t\right)$. By part 1 of assumption 4, $\mathbb{E}{G}^{3\rho }<\infty$. Part 4 of Lemma 8.1 below gives us that $\mathcal{G}$ is Donsker. Since Π satisfies J_[] {∞, Π, L_3ρ(P)} < ∞ to, we have that

$${\mathcal{F}}_1=\left\{{\Omega }^{1/2}\frac{\pi \left(a^t;s^t\right)}{{\mu }^t\left(a^t;s^t\right)}\Phi \left(s^t\right){\left\{\Phi \left(s^t\right)-\gamma \Phi \left(s^{t+1}\right)\right\}}^{⊺}:\pi \in \Pi \right\}$$

satisfies $J_{[]}\left\{\infty ,{\mathcal{F}}_1,L_{3\rho }\left(P\right)\right\}<\infty$ by parts 1 and 2 of Lemma 8.1 below. Moreover, F(a^t, s^t) = ∥Φ(s^t)∥·∥Φ(s^t)−γΦ(s^t+1)∥/μ^t(a^t; s^⊺) is an envelope for ${\mathcal{F}}_1$ with $\mathbb{E}{F}^{3\rho }$ < ∞ by assumption 3 and part 1 of assumption 4. Thus, ${\mathcal{F}}_1$ is Donsker. Let

$${\mathcal{F}}_2=\left\{{\Omega }^{1/2}\frac{\pi \left(a^t;s^t\right)}{{\mu }^t\left(a^t;s^t\right)}{u}^t\Phi \left(s^t\right):\pi \in \Pi \right\}.$$

Similar arguments yield that ${\mathcal{F}}_2$ is Donsker.

Now, let $\widehat{A}\left(\pi \right)=\left\{{\mathbb{E}}_n{f}_{1\pi }:f_{1\pi }\in {\mathcal{F}}_1\right\}$ and $\widehat{B}\left(\pi \right)=\left\{{\mathbb{E}}_n{f}_{2\pi }:f_{2\pi }\in {\mathcal{F}}_2\right\}$. Let $\widehat{A^{\prime }}\left(\pi \right)=\widehat{A}\left(\pi \right)+{\lambda }_n\widehat{A}{\left(\pi \right)}^{-⊺}$. We have that ${\widehat{\theta }}_n^{\pi }=\widehat{A^{\prime }}{\left(\pi \right)}^{-1}\widehat{B}\left(\pi \right)$. Thus,

$$\begin{array}{lll}\sqrt{n}\left({\widehat{\theta }}_n^{\pi }-{\theta }_0^{\pi }\right)\hfill & =\hfill & \sqrt{n}\left\{\widehat{A^{\prime }}{\left(\pi \right)}^{-1}\widehat{B}\left(\pi \right)-\widehat{A^{\prime }}{\left(\pi \right)}^{-1}\widehat{A^{\prime }}\left(\pi \right){\theta }_0^{\pi }\right\}+o_P\left(1\right)\hfill \\ \hfill & =\hfill & \widehat{A^{\prime }}{\left(\pi \right)}^{-1}\sqrt{n}\left\{\widehat{B}\left(\pi \right)-\widehat{A^{\prime }}\left(\pi \right){\theta }_0^{\pi }\right\}+o_P\left(1\right)\hfill \\ \hfill & =\hfill & \widehat{A^{\prime }}{\left(\pi \right)}^{-1}\sqrt{n}\left\{\widehat{B}\left(\pi \right)-\widehat{A}\left(\pi \right){\theta }_0^{\pi }\right\}+\widehat{A^{\prime }}{\left(\pi \right)}^{-1}\sqrt{n}\left\{\widehat{A}\left(\pi \right)-\widehat{A^{\prime }}\left(\pi \right)\right\}{\theta }_0^{\pi }+o_P\left(1\right)\hfill \\ \hfill & =\hfill & \widehat{A^{\prime }}{\left(\pi \right)}^{-1}\sqrt{n}\left\{\widehat{B}\left(\pi \right)-\widehat{A}\left(\pi \right){\theta }_0^{\pi }\right\}+o_P\left(1\right)\hfill \end{array}$$

where o_P(1) doesn’t depend on π, because $\widehat{A^{\prime }}{\left(\pi \right)}^{-1}\stackrel{P}{\to }{w}_1{\left(\pi \right)}^{-1}<\infty$ uniformly over π ∈ Π by assumption 3 and part 1 of assumption 4, ${\text{sup}}_{\pi \in \Pi }\Vert {\theta }_0^{\pi }\Vert <\infty$ by part 1 of this theorem, and $\sqrt{n}\left\{\widehat{A}\left(\pi \right)-\widehat{A^{\prime }}\left(\pi \right)\right\}=-\sqrt{n}{\lambda }_n\widehat{A}{\left(\pi \right)}^{-⊺}=o_P\left(1\right)$ because λ_n = o_P(n^−1/2). Using arguments similar to those in the previous paragraph, one can show that ${\mathcal{F}}_3=\left\{f_{2\pi }-f_{1\pi }\theta :f_{1\pi }\in {\mathcal{F}}_1,f_{2\pi }\in {\mathcal{F}}_2,\pi \in \Pi ,\theta \in B_{\ast }\right\}$ is Donsker, where B_* is any finite collection of elements of ${ℝ}^q$. By part 1 of this theorem, there exists a bounded, closed set B₀ such that ${\theta }_0^{\pi }\in B_0$ for all π ∈ Π. Let ${\mathbb{G}}_n\left(\pi ,\theta \right)=\sqrt{n}\left({\mathbb{E}}_n-\mathbb{E}\right)\left(f_{2\pi }-f_{1\pi }\theta \right)$. Note that

$$\begin{array}{lll}\underset{\pi \in \Pi }{\text{sup}}\Vert {\mathbb{G}}_n\left(\pi ,{\theta }_1\right)-{\mathbb{G}}_n\left(\pi ,{\theta }_2\right)\Vert \hfill & \le \hfill & \underset{\pi \in \Pi }{\text{sup}}\Vert \sqrt{n}\left({\mathbb{E}}_n-\mathbb{E}\right)f_{1\pi }\Vert \cdot \Vert {\theta }_1-{\theta }_2\Vert \hfill \\ \hfill & \le \hfill & R^{\ast }\Vert {\theta }_1-{\theta }_2\Vert ,\hfill \end{array}$$

where R^* = O_P(1) by the Donsker property of ${\mathcal{F}}_1$ and R^* doesn’t depend on π. Thus, ${\mathbb{G}}_n\left(\pi ,\theta \right)$ is stochastically equicontinuous on B₀. Combined with the Donsker property of ${\mathcal{F}}_3$ for arbitrary B_*, we have that the class ${\mathcal{F}}_4=\left\{f_{2\pi }-f_{1\pi }\theta :f_{1\pi }\in {\mathcal{F}}_1,f_{2\pi }\in {\mathcal{F}}_2,\pi \in \Pi ,\theta \in B_0\right\}$ is Donsker. Using Slutsky’s Theorem, Theorem 11.24 of Kosorok (2008), the fact that ${\mathcal{F}}_1$ is Glivenko-Cantelli, and the fact that ${\theta }_0^{\pi }={\left(\mathbb{E}{f}_{1\pi }\right)}^{-1}\mathbb{E}{f}_{2\pi }$, we have that $\sqrt{n}\left({\widehat{\theta }}_0^{\pi }-{\theta }_0^{\pi }\right)=\widehat{A^{\prime }}{\left(\pi \right)}^{-1}{\mathbb{G}}_n\left(\pi ,{\theta }_0^{\pi }\right)⇝w_1{\left(\pi \right)}^{-1}{\mathbb{G}}_0\left(\pi \right)$, in ${\ell }^{\infty }(\Pi )$ where ${\mathbb{G}}_0\left(\pi \right)$ is a mean zero Gaussian process indexed by Π with covariance $\mathbb{E}\left\{{\mathbb{G}}_0\left({\pi }_1\right){\mathbb{G}}_0\left({\pi }_2\right)\right\}=w_0\left({\pi }_1,{\pi }_2\right)$.

Proof of part 3: We have that

$$\begin{array}{lll}\sqrt{n}\left\{{\widehat{V}}_{n,\widehat{\mathcal{R}}}\left(\pi \right)-V_{\mathcal{R}}\left(\pi \right)\right\}\hfill & =\hfill & \sqrt{n}{\mathbb{E}}_n\Phi \left(S^t\right)\left({\widehat{\theta }}_n^{\pi }-{\theta }_0^{\pi }\right)\hfill \\ \hfill & ⇝\hfill & {\nu }^{⊺}{w}_1{\left(\pi \right)}^{-1}{\mathbb{G}}_0\left(\pi \right)\hfill \end{array}$$

in ${\ell }^{\infty }(\Pi )$ by Slutsky’s Theorem.

Proof of Theorem 4.3. Proof of part 1: Following part 3 of Theorem 4.2, we have that ${\text{sup}}_{\beta \in \mathcal{B}}\left|{\widehat{V}}_{n,\widehat{\mathcal{R}}}\left({\pi }_{\beta }\right)-V_{\mathcal{R}}\left({\pi }_{\beta }\right)\right|\stackrel{P}{\to }0$. Combining this with the unique and well separated maximum condition (assumption 6), continuity of $V_{\mathcal{R}}\left({\pi }_{\beta }\right)$ in β, and Theorem 2.12 of Kosorok (2008), yields the result in part 1. Part 2 follows from part 1 of this theorem, part 1 of Theorem 4.2, and the continuous mapping theorem. Part 3 follows from parts 2 and 3 of Theorem 4.2. The proof of part 4 follows standard arguments. ☐

Lemma 8.1. Let $\mathcal{F}$ and $\mathcal{G}$ be function classes with respective envelopes F and G. Let ${\Vert F\Vert }_u={\left(\mathbb{E}{\left|F\right|}^u\right)}^{1/u}$. For any 1 ≤ r, s₁, s₂ ≤ ∞ with $s_1^{-1}+s_2^{-1}=1$,

1. $J_{[]}\left\{\infty ,\mathcal{F}\cdot \mathcal{G},L_r\left(P\right)\right\}\le 2\left({\Vert F\Vert }_{r{s}_1}+{\Vert G\Vert }_{r{s}_2}\right)\left[J_{[]}\left\{\infty ,\mathcal{F},L_{r{s}_1}\left(P\right)\right\}+J_{[]}\left\{\infty ,\mathcal{G},L_{r{s}_2}\left(P\right)\right\}\right]$.

2. $J_{[]}\left\{\infty ,\mathcal{F}+\mathcal{G},L_r\left(P\right)\right\}\le 2\left[J_{[]}\left\{\infty ,\mathcal{F},L_r\left(P\right)\right\}+J_{[]}\left\{\infty ,\mathcal{G},L_r\left(P\right)\right\}\right]$.

3. For any $0\le r\le \infty ,J_{[]}\left\{\infty ,\mathcal{F}\cup \mathcal{G},L_r\left(P\right)\right\}\le \sqrt{\text{log}2}\left({\Vert F\Vert }_r+{\Vert G\Vert }_r\right)+J_{[]}\left\{\infty ,\mathcal{F},L_r\left(P\right)\right\}+J_{[]}\left\{\infty ,\mathcal{G},L_r\left(P\right)\right\}$.

4. If $\mathcal{G}$ is a finite class, $J_{[]}\left\{\infty ,\mathcal{G},L_r\left(P\right)\right\}\le 2{\Vert G\Vert }_r\sqrt{\text{log}\left|\mathcal{G}\right|}$, where $\left|\mathcal{G}\right|$ denotes the cardinality of $\mathcal{G}$.

Proof of Lemma 8.1. Proof of part 1: Let 1 ≤ r, s₁, s₂ ≤ ∞ with $s_1^{-1}+s_2^{-1}=1$ and let (ℓ_F, u_F) and (ℓ_G, u_G) be $L_{r{s}_1}\left(P\right)$ and $L_{r{s}_2}\left(P\right)$ ϵ-brackets, respectively. Choose ℓ_F, ≤ f₁, f₂ ≤ u_F and ℓ_G ≤ g₁, g₂ ≤ u_G and consider the bracket for any f₂g₂ defined by f₁g₁ ± (F|u_G − ℓ_G| + G|u_F − ℓ_F|). Note that f₁g₁+F|u_G−ℓ_G|+G|u_F−ℓ_F|−f₂g₂ ≥ F|u_G−ℓ_G|+G|u_F−ℓ_F|−F|g₁−g₂|−G|f₁−f₂| ≥ 0, because f₂g₂ − f₁g₁ = f₂g₂ + f₂g₁ − f₂g₁ − f₁g₁ ≤ F|g₁ − g₂| + G|f₁ − f₂|. Similarly, f₂g₂ + F|u_G − ℓ_G| + G|u_F − ℓ_F| − f1g1 ≥ 0. Thus, these brackets hold all f₂g₂ for f₂ ∈ (ℓ_F, u_F) and g₂ ∈ (ℓ_G, u_G). Now, ∥F|u_G − ℓ_G| + G|u_F − ℓ_F |∥_r ≤ ∥F∥_rs1ϵ + ∥G∥_rs2ϵ by Minkowski’s inequality and Hölder’s inequality, and it follows that

$$N_{[]}\left\{2ϵ\left({\Vert F\Vert }_{r{s}_1}+{\Vert G\Vert }_{r{s}_2}\right),\mathcal{F}\cdot \mathcal{G},L_r\left(P\right)\right\}\le N_{[]}\left\{ϵ,\mathcal{F},L_{r{s}_1}\left(P\right)\right\}{N}_{[]}\left\{ϵ,\mathcal{G},L_{r{s}_2}\left(P\right)\right\}.$$

Next we note that

$$N_{[]}\left\{ϵ,\mathcal{F}\cdot \mathcal{G},L_r\left(P\right)\right\}\le N_{[]}\left\{\frac{ϵ}{2\left({\Vert F\Vert }_{r{s}_1}+{\Vert G\Vert }_{r{s}_2}\right)},\mathcal{F},L_{r{s}_1}\left(P\right)\right\}{N}_{[]}\left\{\frac{ϵ}{2\left({\Vert F\Vert }_{r{s}_1}+{\Vert G\Vert }_{r{s}_2}\right)},\mathcal{G},L_{r{s}_2}\left(P\right)\right\}$$

and thus

$$\begin{array}{lll}{J}_{[]}\left\{\infty ,\mathcal{F}\cdot \mathcal{G},L_r\left(P\right)\right\}\hfill & \le \hfill & {\displaystyle {\int }_0^{2{\Vert F\Vert }_{r{s}_1}{\Vert G\Vert }_{r{s}_2}}\sqrt{\text{log}{N}_{[]}\left\{\frac{ϵ}{2\left({\Vert F\Vert }_{r{s}_1}+{\Vert G\Vert }_{r{s}_2}\right)},\mathcal{F},L_{r{s}_1}\left(P\right)\right\}}\text{d}ϵ}+{\displaystyle {\int }_0^{2{\Vert F\Vert }_{r{s}_1}{\Vert G\Vert }_{r{s}_2}}\sqrt{\text{log}{N}_{[]}\left\{\frac{ϵ}{2\left({\Vert F\Vert }_{r{s}_1}+{\Vert G\Vert }_{r{s}_2}\right)},\mathcal{G},L_{r{s}_2}\left(P\right)\right\}}\text{d}ϵ}\hfill \\ \hfill & \le \hfill & 2\left({\Vert F\Vert }_{r{s}_1}+{\Vert G\Vert }_{r{s}_2}\right)\left[J_{[]}\left\{\infty ,\mathcal{F},L_{r{s}_1}\left(P\right)\right\}+J_{[]}\left\{\infty ,\mathcal{G},L_{r{s}_2}\left(P\right)\right\}\right].\hfill \end{array}$$

The proof of part 2 follows from Lemma 9.25 part (i) of Kosorok (2008) after a change of variables. Proof of part 3: First note that

$$N_{[]}\left\{ϵ,\mathcal{F}\cup \mathcal{G},L_r\left(P\right)\right\}\le N_{[]}\left\{ϵ,\mathcal{F},L_r\left(P\right)\right\}+N_{[]}\left\{ϵ,\mathcal{G},L_r\left(P\right)\right\},$$

whence it follows that

$$\begin{array}{lll}{J}_{[]}\left\{\infty ,\mathcal{F}\cup \mathcal{G},L_r\left(P\right)\right\}\hfill & =\hfill & {\displaystyle {\int }_0^{2\left({\Vert F\Vert }_r+{\Vert G\Vert }_r\right)}\sqrt{\text{log}{N}_{[]}\left\{ϵ,\mathcal{F}\cup \mathcal{G},L_r\left(P\right)\right\}}\text{d}ϵ}\hfill \\ \hfill & \le \hfill & {\displaystyle {\int }_0^{2\left({\Vert F\Vert }_r+{\Vert G\Vert }_r\right)}\sqrt{\text{log}\left[N_{[]}\left\{ϵ,\mathcal{F},L_r\left(P\right)\right\}+N_{[]}\left\{ϵ,\mathcal{G},L_r\left(P\right)\right\}\right]}\text{d}ϵ}\hfill \\ \hfill & \le \hfill & {\displaystyle {\int }_0^{2\left({\Vert F\Vert }_r+{\Vert G\Vert }_r\right)}\sqrt{\text{log}2+\text{log}{N}_{[]}\left\{ϵ,\mathcal{F},L_r\left(P\right)\right\}+\text{log}{N}_{[]}\left\{ϵ,\mathcal{G},L_r\left(P\right)\right\}}\text{d}ϵ}\hfill \\ \hfill & \le \hfill & {\displaystyle {\int }_0^{2\left({\Vert F\Vert }_r+{\Vert G\Vert }_r\right)}\sqrt{\text{log}2}\text{d}ϵ}+J_{[]}\left\{\infty ,\mathcal{F},L_r\left(P\right)\right\}+J_{[]}\left\{\infty ,\mathcal{G},L_r\left(P\right)\right\},\hfill \end{array}$$

where the second inequality uses the fact that a + b ≤ 2ab for all a, b ≥ 1.

Proof of part 4: If $\mathcal{G}$ is finite, then $N_{[]}\left\{ϵ,\mathcal{G},L_r\left(P\right)\right\}\le \left|\mathcal{G}\right|$. Thus,

$$\begin{array}{lll}{J}_{[]}\left\{\infty ,\mathcal{G},L_r\left(P\right)\right\}\hfill & =\hfill & {\displaystyle {\int }_0^{2{\Vert G\Vert }_r}\sqrt{\text{log}{N}_{[]}\left\{ϵ,\mathcal{G},L_r\left(P\right)\right\}}\text{d}ϵ}\hfill \\ \hfill & \le \hfill & {\displaystyle {\int }_0^{2{\Vert G\Vert }_r}\sqrt{\text{log}\left|G\right|}\text{d}ϵ,}\hfill \end{array}$$

which completes the proof.

Lemma 8.2. Define the class of functions

$$\Pi =\left\{{\pi }_{\tilde{\beta }}\left(a;s\right)=\frac{a_J+{\displaystyle {\sum }_{j=1}^{J-1}{a}_j\text{exp}\left(s_j^{⊺}{\beta }_j\right)}}{1+{\displaystyle {\sum }_{j=1}^{J-1}\text{exp}\left(s_j^{⊺}{\beta }_j\right)}}:\tilde{\beta }={\left({\beta }_1^{⊺},\dots ,{\beta }_{J-1}^{⊺}\right)}^{⊺},\tilde{\beta }\in \mathcal{B}\subset {ℝ}^{p\left(J-1\right)}\right\}$$

for a compact set $\mathcal{B}$ and 2 ≤ J < ∞ where a = (a₁, …, a_J)^⊺. Then, there exists a b₀ < ∞ such that for any 1 ≤ r ≤ ∞, $J_{[]}\left\{\infty ,\Pi ,L_r\left(P\right)\right\}\le b_0{\Vert S\Vert }_r\sqrt{p\left(J-1\right)\pi }$, which is finite whenever ∥S∥_r < ∞. Furthermore, ${\text{sup}}_{\Vert {\tilde{\beta }}_1-{\tilde{\beta }}_2\Vert \le \delta }\mathbb{E}\Vert {\pi }_{{\tilde{\beta }}_1}\left(A;S\right)-{\pi }_{{\tilde{\beta }}_2}\left(A;S\right)\Vert \to 0\text{as}\delta \downarrow 0$.

Proof of Lemma 8.2. For ${\tilde{\beta }}_1,{\tilde{\beta }}_2\in \mathcal{B}$, define $d\left({\tilde{\beta }}_1,{\tilde{\beta }}_2\right)={\text{max}}_{1\le j\le J-1}\Vert {\tilde{\beta }}_{1j}-{\tilde{\beta }}_{2j}\Vert$ and $b_0={\text{sup}}_{{\tilde{\beta }}_1,{\tilde{\beta }}_2\in \mathcal{B}}\Vert {\tilde{\beta }}_1-{\tilde{\beta }}_2\Vert <0$ because $\mathcal{B}$ is compact. By the mean value theorem, for any ${\tilde{\beta }}_1,{\tilde{\beta }}_2\in \mathcal{B}$, there exists a point ${\tilde{\beta }}_{\ast }$ on the line segment between ${\tilde{\beta }}_1$ and ${\tilde{\beta }}_2$ such that

$${\pi }_{{\tilde{\beta }}_1}\left(a;s\right)-{\pi }_{{\tilde{\beta }}_2}\left(a;s\right)=\frac{1}{1+{\displaystyle {\sum }_{j=1}^{J-1}\text{exp}\left(s^{⊺}{\tilde{\beta }}_{\ast j}\right)}}\left[{\displaystyle \sum _{j=1}^{J-1}\left\{a_j-{\pi }_{{\tilde{\beta }}_{\ast }}\left(a;s\right)\right\}\text{exp}\left(s^{⊺}{\tilde{\beta }}_{\ast j}\right)s^{⊺}\left({\tilde{\beta }}_{1j}-{\tilde{\beta }}_{2j}\right)}\right],$$

which implies that

$$\left|{\pi }_{{\tilde{\beta }}_1}\left(a;s\right)-{\pi }_{{\tilde{\beta }}_2}\left(a;s\right)\right|\le \Vert s\Vert d\left({\tilde{\beta }}_1,{\tilde{\beta }}_2\right).$$ (9)

It follows from equation (9) that assumption 7 holds for this particular class of policies. Now, $N_{[]}\left\{2ϵ{\Vert S\Vert }_r,\Pi ,L_r\left(P\right)\right\}\le N\left(ϵ,\mathcal{B},d\right)$ by Theorem 9.23 of Kosorok (2008). Furthermore, $N\left(ϵ,\mathcal{B},d\right)\le \text{max}\left\{{\left(b_0/ϵ\right)}^{p\left(J-1\right)},1\right\}$, and thus

$$\begin{array}{lll}{J}_{[]}\left\{ϵ,\Pi ,L_r\left(P\right)\right\}\hfill & \le \hfill & 2{\Vert S\Vert }_r{\displaystyle {\int }_0^{b_0}\sqrt{p\left(J-1\right)\left\{\text{log}{b}_0+\text{log}\left(1/ϵ\right)\right\}}\text{d}ϵ}\hfill \\ \hfill & \le \hfill & 2{\Vert S\Vert }_r{b}_0\sqrt{p\left(J-1\right)}{\displaystyle {\int }_0^1\sqrt{\text{log}\left(1/ϵ\right)}}\text{d}ϵ\hfill \\ \hfill & =\hfill & 2{\Vert S\Vert }_r{b}_0\sqrt{p\left(J-1\right)}{\displaystyle {\int }_0^{\infty }{u}^{1/2}\text{exp}\left(-u\right)}\text{d}u\hfill \\ \hfill & =\hfill & {\Vert S\Vert }_r{b}_0\sqrt{p\left(J-1\right)\pi },\hfill \end{array}$$

which proves the result.

Additional simulation results

Table 8:

Monte Carlo value estimates for offline simulations with γ = 0.9, $\mathcal{P}$ is the LASSO penalty, and Ω is the identity.

n	T	Linear VL	Polynomial VL	Gaussian VL	GGQ	Observed
25	24	0.123 (0.0773)	0.117 (0.1067)	0.128 (0.0960)	0.025 (0.0330)	−0.005
	36	0.117 (0.0900)	0.120 (0.0933)	0.138 (0.0992)	0.030 (0.0341)	−0.004
	48	0.122 (0.0782)	0.103 (0.1002)	0.141 (0.0878)	0.028 (0.0301)	0.000
50	24	0.109 (0.0727)	0.122 (0.0954)	0.153 (0.0692)	0.028 (0.0321)	−0.005
	36	0.137 (0.0782)	0.127 (0.1061)	0.141 (0.0816)	0.024 (0.0285)	0.003
	48	0.110 (0.0761)	0.127 (0.0860)	0.147 (0.0778)	0.029 (0.0347)	0.000
100	24	0.125 (0.0802)	0.129 (0.0854)	0.164 (0.0609)	0.027 (0.0289)	−0.001
	36	0.151 (0.0739)	0.148 (0.0822)	0.131 (0.0897)	0.025 (0.0356)	−0.002
	48	0.131 (0.0726)	0.132 (0.0814)	0.169 (0.0666)	0.030 (0.0325)	−0.001

Table 9:

Monte Carlo value estimates for offline simulations with γ = 0.9, $\mathcal{P}$ is Euclidean norm, and Ω is the inverse Fisher information.

n	T	Linear VL	Polynomial VL	Gaussian VL	GGQ	Observed
25	24	0.136 (0.0862)	0.118 (0.0925)	0.153 (0.0785)	0.022 (0.0272)	−0.005
	36	0.147 (0.0768)	0.132 (0.1057)	0.124 (0.0909)	0.026 (0.0355)	−0.004
	48	0.128 (0.0897)	0.146 (0.0826)	0.116 (0.1067)	0.020 (0.0317)	0.000
50	24	0.113 (0.0954)	0.129 (0.0964)	0.123 (0.1052)	0.027 (0.0275)	−0.005
	36	0.116 (0.0973)	0.149 (0.0940)	0.152 (0.0798)	0.029 (0.0289)	0.003
	48	0.109 (0.0899)	0.132 (0.0998)	0.124 (0.0932)	0.024 (0.0285)	0.000
100	24	0.167 (0.0652)	0.155 (0.0743)	0.144 (0.0897)	0.025 (0.0291)	−0.002
	36	0.167 (0.0731)	0.153 (0.0988)	0.155 (0.0851)	0.027 (0.0311)	−0.002
	48	0.137 (0.0868)	0.175 (0.0615)	0.148 (0.0978)	0.026 (0.0332)	−0.001

Table 10:

Monte Carlo value estimates for offline simulations with γ = 0.9, $\mathcal{P}$ is the LASSO penalty, and Ω is the inverse Fisher information.

n	T	Linear VL	Polynomial VL	Gaussian VL	GGQ	Observed
25	24	0.123 (0.0750)	0.123 (0.0912)	0.140 (0.0930)	0.025 (0.0301)	−0.005
	36	0.139 (0.0780)	0.110 (0.1013)	0.138 (0.0813)	0.024 (0.0344)	−0.004
	48	0.135 (0.0690)	0.110 (0.1177)	0.143 (0.0718)	0.023 (0.0282)	0.000
50	24	0.118 (0.0705)	0.124 (0.0994)	0.137 (0.0802)	0.030 (0.0287)	−0.006
	36	0.117 (0.0827)	0.123 (0.0972)	0.121 (0.0804)	0.030 (0.0292)	0.003
	48	0.128 (0.0807)	0.113 (0.1085)	0.137 (0.0921)	0.023 (0.0282)	0.000
100	24	0.131 (0.0563)	0.123 (0.1015)	0.167 (0.0472)	0.029 (0.0295)	−0.001
	36	0.132 (0.0735)	0.148 (0.0851)	0.161 (0.0670)	0.029 (0.0334)	−0.002
	48	0.149 (0.0612)	0.137 (0.1003)	0.156 (0.0687)	0.023 (0.0267)	−0.001