Q-LEARNING WITH CENSORED DATA

Abstract

We develop methodology for a multistage-decision problem with flexible number of stages in which the rewards are survival times that are subject to censoring. We present a novel Q-learning algorithm that is adjusted for censored data and allows a flexible number of stages. We provide finite sample bounds on the generalization error of the policy learned by the algorithm, and show that when the optimal Q-function belongs to the approximation space, the expected survival time for policies obtained by the algorithm converges to that of the optimal policy. We simulate a multistage clinical trial with flexible number of stages and apply the proposed censored-Q-learning algorithm to find individualized treatment regimens. The methodology presented in this paper has implications in the design of personalized medicine trials in cancer and in other life-threatening diseases.

1. Introduction

In medical research, dynamic treatment regimes are increasingly used to choose effective treatments for individual patients with long-term patient care. A dynamic treatment regime (or similarly, policy) is a set of decision rules for how the treatment should be chosen at each decision time-point, depending on both the patient’s medical history up to the current time-point and the previous treatments. Note that although the same set of decision rules are applied to all patients, the choice of treatment at a given time-point may differ, depending on the patient’s medical state. Moreover, the patient’s treatment plan is not known at the beginning of a dynamic regime, since it may depend on subsequent time-varying variables that may be influenced by earlier treatments and response to treatment. An optimal treatment regime is a set of treatment choices that maximizes the mean response of some clinical outcome at the end of the final time interval (see, for example, Murphy, 2003; Robins, 2004; Moodie et al., 2007).

We consider the problem of finding treatment regimes that lead to longer survival times, where the number of treatments is flexible and where the data is subject to censoring. This type of framework is natural for cancer applications, where the initiation of the next line of therapy depends on the disease progression and thus the number of treatments is flexible. In addition, data are subject to censoring since patients can drop out during the trial. For example, in advanced non-small cell lung cancer (NSCLC), patients receive one to three treatment lines. The timing of the second and third lines of treatment is determined by the disease progression and by the ability of patients to tolerate therapy (Stinchcombe and Socinski, 2008; Krzakowski et al., 2010). We focus on mean survival time restricted to a specific interval, since in a limited-time study, censoring prevents reliable estimation of the unrestricted mean survival time (see discussion in Karrison, 1997; Zucker, 1998; Chen and Tsiatis, 2001, see also Wahed and Tsiatis, 2006 in the context of sequential decision problems and see Robins et al., 2008 for an alternative approach).

Finding an optimal policy for survival data poses many statistical challenges. We enumerate four. First, one needs to incorporate information accrued over time into the decision rule. Second, one needs to avoid treatments which appear optimal in the short term but may lead to poor final outcome in the long run. Third, the data is subject to censoring since some of the patients may be lost to follow-up and the final outcome of those who reached the end of the study alive is unknown. Fourth, the number of decision points (i.e., treatments) and the timing of these decision points can be different for different patients. This follows since the number of treatments and duration between treatments may depend on the medical condition of the patient. In addition, in the case of patient’s death, treatment is stopped. The first two challenges are shared with general multistage decision optimization (Lavori and Dawson, 2004; Moodie et al., 2007). The latter two arise naturally in the context of optimizing survival time, but are applicable to other scenarios as well. Developing valid methodology for estimating dynamic treatment regimes in this flexible timing set-up is crucial for applications in cancer and in other diseases where such structure is the norm and appropriate existing methods are unavailable.

One of the primary tools used in developing dynamic treatment regimes is Q-learning (Murphy et al., 2006; Zhao et al., 2009; Laber et al., 2010; Zhao et al., 2010). Q-learning (Watkins, 1989; Watkins and Dayan, 1992), which is reviewed in Section 2, is a reinforcement learning algorithm. Since we do not assume that the problem is Markovian, we present a version of Q-learning that uses backward recursion. The backward recursion used by Q-learning addresses the first two challenges posed above: It enables both accrual of information and incorporation of long-term treatment effects. However, when the number of stages is flexible, and censoring is introduced, it is not clear how to implement backward recursion. Indeed, finding the optimal treatment at the last stage is not well defined, since the number of stages is patient-dependent. Also, it is not clear how to utilize the information regarding censored patients.

In this paper we present a novel Q-learning algorithm that takes into account the censored nature of the observations using inverse-probability-of-censoring weighting (see, Robins et al., 1994, see also Wahed and Tsiatis, 2006; Robins et al., 2008 in the context of sequential decision problems). We provide finite sample bounds on the generalization error of the policy learned by the algorithm, i.e., bounds on the average difference in expected survival time between the optimal dynamic treatment regime and the dynamic treatment regime obtained by the proposed Q-learning algorithm. We also present a simulation study of a sequential-multiple-assignment randomized trial (SMART) (see Murphy, 2005a, and references therein) with flexible number of stages depending on disease progression and failure event timing. We demonstrate that the censored-Q-learning algorithm proposed here can find treatment strategies tailored to each patient which are better than any fixed-treatment sequence. We also demonstrate the result from ignoring censored observations.

One general contribution of the paper is the development of a methodology for solving backward recursion when the number and timing of stages is flexible. As mentioned previously, this is crucial for applications but has not been addressed previously. In Section 4 we present an auxiliary multistage decision problem that has a fixed number of stages. Since the number of stages is fixed for the auxiliary problem, backward recursion can be used in order to estimate the decision policy. We then show how to translate the original problem to the auxiliary one and obtain the surprising conclusion that results obtained for the auxiliary problem can be translated into results regarding the original problem with flexible number and timing of stages.

An additional contribution of the paper is the universal consistency proof for the algorithm performance. Universal consistency of an algorithm means that for every distribution function on the sample space, the expected loss of the function learned by the algorithm converges in probability to the infimum over all measurable functions of the expected loss (see, for example, Steinwart and Chirstmann, 2008 ). In Section 6 we prove that when the optimal Q-functions belong to the corresponding approximation spaces considered by the algorithm, the algorithm is universally consistent. The proof presented here is algorithm-specific, but the tools used in the proof are widely applicable for universal consistency proofs when the data are subject to censoring (see, for example, Goldberg et al., 2011 ). While other learning algorithms were suggested for survival data (see, for example, Biganzoli et al., 1998; Shivaswamy et al., 2007; Shim and Hwang, 2009, see also Zhao et al., 2010 in the context of a multistage decision problem), we are not aware of any other universal consistency proof for survival data.

The paper is organized as follows. In Section 2 we review the Q-learning algorithm and discuss the challenges for adapting the Q-leaning methodology for a framework with flexible number of stages and censored data. We also review existing methods for finding optimal policies. Definitions and notation are presented in Section 3. The auxiliary problem is presented in Section 4. The censored-Q-learning algorithm is presented in Section 5. The main theoretical results are presented in Section 6. In Section 7 we present a multistage-randomized-trial simulation study. Concluding remarks appear in Section 8. Supplementary proofs are provided in Appendix A. A description of and link to the code and data sets used in Section 7 appear in Supplement A

2. Q-learning

2.1. Reinforcement Learning

Reinforcement learning is a methodology for solving multistage decision problems. It involves recording sequences of actions, statistically estimating the relationship between these actions and their consequences and then choosing a policy (i.e., a set of decision rules) that approximates the most desirable consequence based on the statistical estimation. A detailed introduction to reinforcement learning can be found in Sutton and Barto (1998).

In the medical context of long-term patient care, the reinforcement learning setting can be described as follows. For each patient, the stages correspond to clinical decision points in the course of the patient’s treatment. At these decision points, actions (e.g., treatments) are chosen, and the state of the patient is recorded. As a consequence of a patient’s treatment, the patient receives a (random) numerical reward.

More formally, consider a multistage decision problem with T decision points. Let S_t be the (random) state of the patient at stage t ∈ {1, …, T +1}and let S_t = {S₁,…, S_t} be the vector of all states up to and including stage t. Similarly, let A_t be the action chosen in stage t, and let A_t = {A₁, …, A_t} be the vector of all actions up to and including stage t. We use the corresponding lower case to denote a realization of these random variables and random vectors. Let the random reward be denoted R_t = r(S_t, A_t, S_t+1), where r is a (unknown) time-dependent deterministic function of all states up to stage t+1 and all past actions up to stage t. A trajectory is defined as a realization of (S_T+1, A_T, R_T). Note that we do not assume that the problem is Markovian. In the medical context example, a trajectory is a record of all the patient covariates at the different decision points, the treatments that were given, and the medical outcome in numerical terms.

We define a policy, or similarly, a dynamic treatment regime, to be a set of decision rules. More formally, define a policy π to be a sequence of deterministic decision rules, {π₁, …, π_T}, where for every pair (s_t, a_t−1), the output of the t-th decision rule, π_t(s_t, a_t−1), is an action. Our goal is to find a policy that maximizes the expected sum of rewards. The Bellman equation (Bellman, 1957) characterizes the optimal policy π* as one that satisfies the following recursive relation:

$${\pi }_t^{\ast }({\mathit{s}}_t,{\mathit{a}}_{t-1})=\underset{a_t}{\text{argmax}}E[R_t+V_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t)|{\mathit{S}}_t={\mathit{s}}_t,{\mathit{A}}_t={\mathit{a}}_t],$$ (1)

where the value function

$$V_{t+1}^{\ast }({\mathit{s}}_{t+1},{\mathit{a}}_t)=E_{{\pi }^{\ast }}\left[\sum _{i=t+1}^T{R}_i|{\mathit{S}}_{t+1}={\mathit{s}}_{t+1},{\mathit{A}}_t={\mathit{a}}_t\right]$$ (2)

is the expected cumulative sum of rewards from stage t + 1 to stage T, where the history up to stage t + 1 is given by {s_t+1, a_t}, and when using the optimal policy π* thereafter.

Finding a policy that leads to a high expected cumulative reward is the main goal of reinforcement learning. Naively, one could learn the transition distribution functions and the reward function using the observed trajectories, and then solve the Bellman equation recursively. However, this approach is inefficient both computationally and memory-wise. In the following section, we introduce the Q-learning algorithm, which requires less memory and less computation.

2.2. Q-learning

Q-learning (Watkins, 1989) is an algorithm for solving reinforcement learning problems. It is claimed by Sutton and Barto to be one of the most important breakthroughs in reinforcement learning (Sutton and Barto, 1998, Section 6.5). Q-learning uses backward recursion to compute the Bellman equation without the need to know the full dynamics of the process.

More formally, we define the optimal time-dependent Q-function

$$Q_t^{\ast }({\mathit{s}}_t,{\mathit{a}}_t)=E[R_t+V_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t)\mid {\mathit{S}}_t={\mathit{s}}_t,{\mathit{A}}_t={\mathit{a}}_t].$$

Note that $V_t^{\ast }({\mathit{s}}_t,{\mathit{a}}_{t-1})={max}_{a_t}{Q}_t^{\ast }({\mathit{s}}_t,{\mathit{a}}_t)$, and thus

$$Q_t^{\ast }({\mathit{s}}_t,{\mathit{a}}_t)=E\left[R_t+\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})|{\mathit{S}}_t={\mathit{s}}_t,{\mathit{A}}_t={\mathit{a}}_t\right].$$ (3)

In order to estimate the optimal policy, one first estimates the Q-functions backwards through time t = T, T − 1, …, 1 and obtains a sequence of estimators {Q̂_T …, Q̂₁}. The estimated policy is given by

$${\widehat{\pi }}_t({\mathit{s}}_t,{\mathit{a}}_{t-1})=\underset{a_t}{\text{argmax}}{\widehat{Q}}_t({\mathit{s}}_t,{\mathit{a}}_{t-1},a_t).$$ (4)

In the next section we discuss the difficulties in applying the Q-learning methodology when trajectories are subject to censoring and the number of stages is flexible.

2.3. Challenges With Flexible Number of Stages and Censoring

As discussed in the introduction, our goal is to develop a Q-learning algorithm that can handle a flexible number of stages and that takes into account the censored nature of the observations. We face two main challenges. First, recall that the estimation of the Q-functions in (3) is done recursively, starting from the last stage backward. Thus, when the number of stages is flexible, it is not clear how to perform the base step of the recursion. Second, due to censoring, some of the trajectories may be incomplete. Incorporating the data of a censored trajectory is problematic: even when the number of stages is fixed, the known number of stages for a censored trajectory may be less than the number of stages in the multistage problem. Moreover, the reward is not known for the stage at which censoring occurs.

2.4. Review of Existing Approaches

Finding optimal policies or optimal treatment regimes has been discussed extensively in other work. We discuss shortly some additional work that is related to the approach taken here. However, we are not aware of any other existing approaches that address simultaneously both censoring and flexible number of stages.

The approach closest to our proposal is the censored-Q-learning algorithm of Zhao et al. (2010). Zhao et al. considered a Q-learning algorithm for censored data based on support vector regression adjusted for censoring with fixed number of stages. A simulation study was performed to demonstrate the algorithm’s performance; however, the theoretical properties of this algorithm were not evaluated.

A general approach for finding optimal polices that uses backward recursion was studied by Murphy (2003) and Robins (2004) in the semiparametric context, and by Murphy (2005b) in the nonparametric context. These works do not treat flexible number of stages or censoring, and cannot be applied to the framework considered here without some adjustments.

Another approach for finding optimal policies was studied by Orellana et al. (2010), (see also Laan and Petersen, 2007; Robins et al., 2008). Orellana et al. consider dynamic regime marginal structural mean models (Robins, 1999). In this approach, for each regime, one considers all trajectories that comply to the regime up to some point. The trajectories are then censored at the first time-point at which they do not comply to the regime. The contribution of the non-compliant trajectories is redistributed among compliant trajectories that have the same covariate and treatment history, using the inverse-probability-of-censoring weighting. Advantages and disadvantages of this approach compared to the backward recursion approach mentioned above are discussed in Robins et al. (2008, Section 5). We note that it is assumed in their approach that the length of each stage is fixed, an assumption we do not require.

This general issue is also related to the analysis of two-stage randomized trials involving right-censored data studied in a series of papers including Lunceford et al. (2002); Wahed and Tsiatis (2006); Wahed (2009); Miyahara and Wahed (2010). The authors use inverse-probability-of-censoring to correct for censoring. See also Thall et al. (2007) that considers analysis of two-stage randomized trials with interval censoring. However, the main focus of these works is in finding the best regime from a finite number of optional regimes, as opposed to the individualized-treatment policies addressed in our proposal.

3. Preliminaries

In this section we present definitions and notation which will be used in the paper.

Let T be the maximal number of decision time-points for a given multistage time-dependent decision problem. Note that the number of stages for different observations can be different. For each t = 1,…,T, the state S_t is the pair S_t = (Z_t, R_t−1), where Z_t is either a vector of covariates describing the state of the patient at beginning of stage t or Z_t = ∅. Z_t = ∅ indicates that a failure event happened during the t-th stage which has therefore reached a terminal state. R_t−1 is the length of the interval between decision time-points t − 1 and t, where we denote R₀ ≡ 0. Although in the usual Q-learning context ${\sum }_{j=1}^t{R}_j$ is the sum of rewards up to and including stage t, in our context it is more useful to think of this sum as the total survival time up to and include stage t. Let A_t be an action chosen at decision time t, where A_t takes its values in a finite discrete space .

The model assumes that observations are subject to censoring. Let C be a censoring variable and let S_C(x) = P (C ≥ x) be its survival function. We assume that censoring is independent of both covariates and failure time. We assume that C takes its values in the segment [0, τ] where τ < ∞ and that S_C(τ) > K_min > 0. Let δ_t be an indicator with δ_t = 1 if no censoring event happened before the t + 1-th decision time-point. Note that δ_t−1 = 0 ⇒ δ_t = 0.

Remark 3.1

Note that for a censoring variable, we define the survival function S_C(x) as P (C ≥ x) rather than the usual P(C > x). This is because given a failure time x, we are interested in the probability P(C ≥ x). However, to avoid complications that are not of interest to the main results of this paper, we assume that the probability of simultaneous failure and censoring is zero (see, for example, Satten and Datta, 2001).

The inclusion of failure times in the model affects the trajectory structure. Usually, a trajectory is defined as a (2T +1)-length sequence {S₁, A₁, S₂, …, A_T, S_T+1}. However, in our context, if a failure event occurs before decision-time-point T, the trajectory will not be of full length. Denote by T̄ the (random) number of stages for the individual (T̄ ≤ T). Due to the censoring, the trajectories themselves are not necessarily fully observed. Assume that a censoring event occurred during stage t. Note that this means that δ_t−1 = 1 while δ_t = 0 and that $C<{\sum }_{i=1}^t{R}_i$. In this case the observed trajectories have the following structure: {S₁, A₁, S₂, …, A_t} and C is also observed.

We now discuss the distribution of the observed trajectories. Assume that n trajectories are sampled at random according to a fixed distribution de-noted by P₀. The distribution P₀ is composed of the unknown distribution of each S_t conditional on (S_t−1, A_t−1) (denoted by {f₁, …, f_T}) and an exploration policy that generates the actions. Denote the exploration policy by p = {p₁, …, p_T} where the probability that action a is taken given history {S_t, A_t−1} is p_t(a|S_t, A_t−1). We assume that p_t(a|s_t, a_t−1) ≥ L⁻¹ for every action a ∈ and for each possible value (s_t, a_t−1), where L ≥ 1 is a constant. The likelihood (under P₀) of the trajectory {s₁, a₁, s₂, …, a_t, s_t̄+1} is

$$f_1(s_1)p_1(a_1\mid s_1)\prod _{j=2}^{\overline{t}}(f_j(s_j\mid {\mathit{s}}_{j-1},{\mathit{a}}_{j-1})p_j(a_j\mid {\mathit{s}}_j,{\mathit{a}}_{j-1}))f_{\overline{t}+1}(s_{t+1}\mid {\mathit{s}}_{\overline{t}},{\mathit{a}}_{\overline{t}}).$$

We denote expectations with respect to the distribution P₀ by E₀. The survival time with respect to the distribution P₀ is denoted by $P_0({\sum }_{j=1}^{\overline{T}}{R}_j>x)$. We assume that G(τ) > G_min > 0, i.e., that there is a positive probability that the survival time is greater than τ.

We define policy π to be a sequence of deterministic decision rules, {π₁, …, π_T}, where for every non-terminating pair (s_t, a_t−1), the output of the t-th decision rule, π_t(s_t, a_t−1), is an action. Let the distribution P_0,π denote the distribution of a trajectory for which the policy π is used to generate the actions. The likelihood (under P_{0, π}) of the trajectory, {s₁, a₁, s₂, …, a_t, s_t̄+1} is

$$f_1(s_1)1_{\pi (s_1)=a_1}\prod _{j=2}^{\overline{t}}(f_j(s_j\mid {\mathit{s}}_{j-1},{\mathit{a}}_{j-1})1_{{\pi }_j({\mathit{s}}_j,{\mathit{a}}_{j-1})=a_j}))f_{\overline{t}+1}(s_{\overline{t}+1}\mid {\mathit{s}}_{\overline{t}},{\mathit{a}}_{\overline{t}}).$$

Our goal is to find a policy that maximizes the expected rewards. Since with probability one C ≤ τ, the maximum observed survival time is less than or equal to τ. Thus we try to maximize the truncated-by-τ expected survival time. Formally, we look for a policy π̂ that approximates the maximum over all deterministic policies of the following expectation:

$$E_{0,\pi }\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau \right],$$

where E_0,π is the expectation with respect to P_0,π and a ∧ b = min{a, b}.

4. The Auxiliary Problem

In this section we construct an auxiliary Q-learning model for our original problem. The modified trajectories of the construction are of fixed length T, and the modified sum of rewards is less than or equal to τ. We then show how results obtained for the auxiliary problem can be translated into results regarding the original problem.

For the auxiliary problem, we complete all trajectories to full length in the following way. Assume that a failure time occurred at stage t < T. In that case the trajectory up to S_t+1 is already defined. Write $S_j^{\prime }=S_j$ for 1 ≤ j ≤ t + 1 and $A_j^{\prime }=A_j$ for 1 ≤ j ≤ t. For all t + 1 < j ≤ T + 1 set S_j = (∅, 0) and for all t + 1 ≤ j ≤ T draw A_j uniformly from .

We also modify trajectories with overall survival time greater than τ in the following way. Assume that t is the first index for which ${\sum }_{i=1}^t{R}_i\ge \tau$. For all j ≤ t, write $S_j^{\prime }=S_j$ and $A_j^{\prime }=A_j$. Write $R_t^{\prime }=\tau -{\sum }_{i=1}^{t-1}{R}_i$ and assign $Z_{t+1}^{\prime }\equiv \varnothing$ and thus the modified state $S_{t+1}^{\prime }=(\varnothing ,R_t^{\prime })$. If t < T, then for all t + 1 < j ≤ T + 1 set S_j = (∅, 0) and for all t + 1 ≤ j ≤ T draw $A_j^{\prime }$ uniformly from . The modified trajectory is given by the sequence { $S_1^{\prime },A_1^{\prime },S_2^{\prime },\dots ,A_T^{\prime },S_{T+1}^{\prime }$}. Note that trajectories with fewer than 2T + 1 entries and for which ${\sum }_{i=1}^t{R}_i\ge \tau$ are modified twice.

The n modified trajectories are distributed according to the fixed distribution P which can be obtained from P₀. This distribution is composed of the unknown distribution of each $S_t^{\prime }$ conditional on ( ${\mathit{S}}_{t-1}^{\prime },{\mathit{A}}_{t-1}^{\prime }$), denoted by { $f_1^{\prime },\dots ,f_{T+1}^{\prime }$}, and exploration policy p′. The conditional distribution $f_1^{\prime }$ equals f₁, and for 2 ≤ t ≤ T + 1,

$$f_t^{\prime }({\mathit{s}}_t^{\prime }\mid {\mathit{s}}_{t-1}^{\prime },{\mathit{a}}_{t-1}^{\prime })=\{\begin{array}{cc}{f}_t((z_t^{\prime },r_t^{\prime })\mid {\mathit{s}}_{t-1}^{\prime },{\mathit{a}}_{t-1}^{\prime })& z_{t-1}^{\prime }\ne \varnothing ,{\sum }_{i=1}^t{r}_i^{\prime }<\tau \\ {\int }_{G_{z_t^{\prime }}}{f}_t((z_t^{\prime },r_t)\mid {\mathit{s}}_{t-1}^{\prime },{\mathit{a}}_{t-1}^{\prime }){\text{d}r}_t& z_{t-1}^{\prime }\ne \varnothing ,{\sum }_{i=1}^t{r}_i^{\prime }=\tau \\ 1_{{\mathit{s}}_t^{\prime }=(\varnothing ,0)}& z_{t-1}^{\prime }=\varnothing \end{array},$$ (5)

where $G_{z_t^{\prime }}=\{(z_t^{\prime },r_t):{\sum }_{i=1}^t{r}_i\ge \tau \}$ and 1_A is 1 if A is true and is 0 otherwise. The exploration policy p′ agrees with p on every pair (S_t, A_t−1) for which Z_t ≠ ∅ and draws A_t uniformly from whenever Z_t = ∅. The likelihood (under P) of the modified trajectory, { $s_1^{\prime },a_1^{\prime },s_2^{\prime },\dots ,a_T^{\prime },s_{T+1}^{\prime }$}, is

$$f_1^{\prime }(s_1^{\prime })p_1(a_1^{\prime }\mid s_1^{\prime })\prod _{t=2}^T(f_t^{\prime }(s_t^{\prime }\mid {\mathit{s}}_{t-1}^{\prime },{\mathit{a}}_{t-1}^{\prime })p_t(a_t^{\prime }\mid {\mathit{s}}_t^{\prime },{\mathit{a}}_{t-1}^{\prime }))f_{T+1}^{\prime }(s_{T+1}^{\prime }\mid {\mathit{s}}_T^{\prime },{\mathit{a}}_T^{\prime }).$$

Denote expectations with respect to the distribution P by E.

Let π be a policy for the original problem. We define a version of the policy π′ for the auxiliary problem in the following way. For any state ( ${\mathit{s}}_t^{\prime },{\mathit{a}}_{t-1}^{\prime }$) for which $z_t^{\prime }\ne \varnothing$, the same action is chosen. For any state ( ${\mathit{s}}_t^{\prime },{\mathit{a}}_{t-1}^{\prime }$) for which $z_t^{\prime }=\varnothing$, a fixed action a_t ∈ is chosen, w.o.l.g., let a_o be chosen. For the auxiliary problem, we say that two policies ${\pi }_a^{\prime }$ and ${\pi }_b^{\prime }$ are equivalent if ${\pi }_a^{\prime }({\mathit{s}}_t^{\prime },{\mathit{a}}_{t-1}^{\prime })={\pi }_b^{\prime }({\mathit{s}}_t^{\prime },{\mathit{a}}_{t-1}^{\prime })$ for every ( ${\mathit{s}}_t^{\prime },{\mathit{a}}_{t-1}^{\prime }$) for which $z_t^{\prime }\ne \varnothing$. We denote both the original policy and any modified version of it by π whenever it is clear from the context which policy is considered. Similarly, we omit the prime from states and actions in the auxiliary problem whenever there is no reason for confusion.

Let P_π be the distribution in the auxiliary problem where actions are chosen according to π. The likelihood under P_π of the trajectory {s₁, a₁, s₂, …, a_T, s_T+1} is

$$f_1^{\prime }(s_1)1_{{\pi }_1(s_1)=a_1}\prod _{t=2}^T(f_t^{\prime }(s_t\mid {\mathit{s}}_{t-1},{\mathit{a}}_{t-1})1_{{\pi }_j({\mathit{s}}_t,{\mathit{a}}_{t-1})=a_t}))f_{T+1}^{\prime }(s_{T+1}\mid {\mathit{s}}_T,{\mathit{a}}_T).$$

Denote expectations with respect to the distribution P_π by E_π.

We now define the value functions and the Q-functions for policies in the auxiliary model. For any auxiliary policy π define its corresponding value function V_π. Given an initial state s₁, V_π(s₁) is the expected truncated-by-τ survival time when the initial state is s₁ and the actions are chosen according to the policy π. Formally $V_{\pi }(s_1)=E_{\pi }[{\sum }_{i=1}^T{R}_t\mid S_1=s_1]$ where the truncation takes place since the expectation is taken with respect to the distribution of the modified trajectories. The stage-t value function for the auxiliary policy π, V_π,t(s_t, a_t−1), is the expected (truncated) remaining survival time from the t-th decision time-point, given the trajectory (s_t, a_t−1), and when following the policy π thereafter. Note, that given s_t, the survival time up to the beginning of stage t is known, and thus truncation ensures that the overall survival time is less than or equal to τ. Formally $V_{\pi ,t}({\mathit{s}}_t,{\mathit{a}}_{t-1})=E_{\pi }[{\sum }_{i=t}^T{R}_i\mid {\mathit{S}}_t={\mathit{s}}_t,{\mathit{A}}_t={\mathit{a}}_t]$.

The stage-t Q-function for the auxiliary policy π is the expected remaining (truncated) survival time, given that the state is (s_t, a_t−1), that a_t is chosen at stage t, and that π is followed thereafter. Formally,

$$Q_{\pi ,t}({\mathit{s}}_t,{\mathit{a}}_t)=E[R_t+V_{\pi ,t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t)\mid {\mathit{S}}_t={\mathit{s}}_t,{\mathit{A}}_t={\mathit{a}}_t].$$

The optimal value function $V_t^{\ast }({\mathit{s}}_t,{\mathit{a}}_{t-1})$ and the optimal Q-function $Q_t^{\ast }({\mathit{s}}_t,{\mathit{a}}_t)$ are define by (2) and (3), respectively.

The following lemma relates the values of the value function V_π in the auxiliary problem to the expected truncated-by-τ survival time for a policy π in the original problem.

Lemma 4.1

Let Π be the collection of all policies in the original problem. Then for all π ∈ Π, the following equalities hold true:

$$V_{\pi }(s_o)=E_{0,\pi }\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau |S_1=s_o\right],$$ (6)

$$V^{\ast }(s_o)=\underset{\pi \in \mathrm{\Pi }}{max}{E}_{0,\pi }\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau |S_1=s_o\right].$$ (7)

Proof

We start by decomposing the expectations depending on both the terminal stage and whether the sum of rewards is greater than or equal to τ.

Define

$$\begin{array}{l}{F}_t=\{\{s_o,a_1,\dots ,s_{t+1}\}:{\sum }_{i=1}^t{r}_i<\tau ,z_{t+1}=\varnothing \},\\ G_t=\{\{s_o,a_1,\dots ,s_{k+1}\}:t=min\{j:\sum _{i=1}^j{r}_i\ge \tau \},\text{and}k=T\text{or}{z}_{k+1}=\varnothing \},\\ F_t^{\prime }=\{({\mathit{s}}_{T+1}^{\prime },{\mathit{a}}_T^{\prime }):({\mathit{s}}_{t+1}^{\prime },{\mathit{a}}_t^{\prime })\in F_t,\{a_{t+1}^{\prime },\dots ,s_{T+1}\}=\{a_o,(\varnothing ,0),\dots ,(\varnothing ,0)\}\},\\ G_t^{\prime }=\left\{\begin{array}{cc}({\mathit{s}}_{T+1}^{\prime },{\mathit{a}}_T^{\prime }):& \begin{array}{l}({\mathit{s}}_t^{\prime },{\mathit{a}}_t^{\prime })\text{is}\text{a}\text{beginning}\text{of}\text{sequence}\text{in}{G}_t,\\ \{s_{t+1}^{\prime },a_{t+1}^{\prime },\dots ,s_{T+1}\}=\{(\varnothing ,\tau -{\sum }_{j=1}^{t-1}{r}_j),a_o,\dots ,(\varnothing ,0)\}\end{array}\end{array}\right\}.\end{array}$$

Denote

$${\mathit{f}}_{t,\pi }({\mathit{s}}_t,{\mathit{a}}_{t-1})=f_1(s_1)\left[1_{\pi (s_1)=a_1}\right]\prod _{j=2}^{t-1}(f_j(s_j\mid {\mathit{s}}_{j-1},{\mathit{a}}_{j-1})1_{{\pi }_j({\mathit{s}}_j,{\mathit{a}}_{j-1})=a_j}))f_t(s_t\mid {\mathit{s}}_{t-1},{\mathit{a}}_{t-1})$$

and similarly $f_{t,\pi }^{\prime }$.

Note that

$$E_{0,\pi }\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau |S_1=s_o\right]=\sum _{t=1}^T{\int }_{F_t}\left(\sum _{i=1}^t{r}_i\right){\mathit{f}}_{t+1,\pi }({\mathit{s}}_{t+1},{\mathit{a}}_t)\text{d (}{\mathit{s}}_{t+1},{\mathit{a}}_t\text{)}+\tau \sum _{t=1}^T{P}_{0,\pi }(G_t),$$ (8)

and

$$V_{\pi }(s_o)=\sum _{t=1}^T{\int }_{F_t^{\prime }}\left(\sum _{i=1}^T{r}_i\right){\mathit{f}}_{T+1,\pi }^{\prime }({\mathit{s}}_{T+1},{\mathit{a}}_T)\text{d (}{\mathit{s}}_{T+1},{\mathit{a}}_T\text{)}+\tau \sum _{t=1}^T{P}_{\pi }(G_t^{\prime }).$$ (9)

Note that

$$\begin{array}{l}{\int }_{F_t}\left(\sum _{i=1}^t{r}_i\right){\mathit{f}}_{t+1,\pi }({\mathit{s}}_{t+1},{\mathit{a}}_t)\text{d}({\mathit{s}}_{t+1},{\mathit{a}}_t)={\int }_{F_t}\left(\sum _{i=1}^t{r}_i\right){\mathit{f}}_{t+1,\pi }^{\prime }({\mathit{s}}_{t+1},{\mathit{a}}_t)\text{d}({\mathit{s}}_{t+1},{\mathit{a}}_t)\\ ={\int }_{F_t^{\prime }}\left(\sum _{i=1}^T{r}_i\right){\mathit{f}}_{T+1,\pi }^{\prime }({\mathit{s}}_{T+1},{\mathit{a}}_T)\text{d}({\mathit{s}}_{T+1},{\mathit{a}}_T)\end{array}$$ (10)

where the first equality follows from (5) and the second follows since there is a one to one correspondence between trajectories in F_t and $F_t^{\prime }$, and by construction, for each such trajectory in $F_t^{\prime }$ we have ${\sum }_{i=t+1}^T{r}_i=0$ and

$$\left[1_{{\pi }_{t+1}({\mathit{s}}_{t+1},{\mathit{a}}_t)=a_o}\right]\prod _{j=t+2}^T(f_j^{\prime }(s_j\mid {\mathit{s}}_{j-1},{\mathit{a}}_{j-1})1_{{\pi }_j({\mathit{s}}_j,{\mathit{a}}_{j-1})=a_o}))f_{T+1}(s_{T+1}\mid {\mathit{s}}_T,{\mathit{a}}_T)=1.$$

Similarly we show that $P_{0,\pi }(G_t)=P_{\pi }(G_t^{\prime })$. Denote by Ĝ_t the set of all sequences (s_t, a_t) which are the beginning part of some trajectory in G_t. Note that

$$\begin{array}{l}{P}_{0,\pi }(G_t)={\int }_{{\widehat{G}}_t}{\mathit{f}}_t({\mathit{s}}_t,{\mathit{a}}_{t-1})\left[1_{{\pi }_t({\mathit{s}}_t,{\mathit{a}}_{t-1})=a_t}\right]{\int }_{\{s_{t+\text{1}}:{\sum }_{i=1}^t{r}_i\ge \tau \}}{f}_{t+1}(s_{t+1}\mid {\mathit{s}}_t,{\mathit{a}}_t)\text{d}({\mathit{s}}_{t+1})\text{d (}{\mathit{s}}_t,{\mathit{a}}_t\text{)}\\ ={\int }_{{\widehat{G}}_t}{\mathit{f}}_t^{\prime }({\mathit{s}}_t,{\mathit{a}}_{t-1})\left[1_{{\pi }_t({\mathit{s}}_t,{\mathit{a}}_{t-1})=a_t}\right]{\int }_{\{s_{t+\text{1}}:{\sum }_{i=1}^t{r}_i\ge \tau \}}{f}_{t+1}^{\prime }(s_{t+1}\mid {\mathit{s}}_t,{\mathit{a}}_t)\text{d}({\mathit{s}}_{t+1})\text{d (}{\mathit{s}}_t,{\mathit{a}}_t\text{)}\\ ={\int }_{G_t^{\prime }}{\mathit{f}}_{T+1}^{\prime }({\mathit{s}}_{T+1},{\mathit{a}}_t)\text{d}({\mathit{s}}_{T+1},{\mathit{a}}_t)=P_{\pi }(G_t^{\prime }),\end{array}$$ (11)

where the second equality follows from (5) and the third equality follows from the construction of $G_t^{\prime }$.

The first assertion of the lemma, namely, equation (6), follows by substituting the right hand side of the equalities (10) and (11) in (8) for each t and comparing to (9).

The second assertion, (7), is proven by maximizing both sides of (6) over all policies. Note that the maximization is taken over two different sets since each policy in the original problem has an equivalent class of polices in the auxiliary problem. However, since V_π is the same for all policies in the same equivalence class, the result follows.

5. The Censored-Q-Learning Algorithm

We now present the proposed censored-Q-learning algorithm. As discussed before, we are looking for a policy π̂ that approximates the maximum over all deterministic policies of the following expectation:

$$E_{0,\pi }\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau \right].$$

We find this policy in three steps. First, we map our problem to the corresponding auxiliary problem. Then we approximate the functions { $Q_1^{\ast },\dots ,Q_T^{\ast }$} using backward recursion based on (3) and obtain the functions {Q̂₁, …, Q̂_T}. Finally, we define π̂ by maximizing Q̂_t(s_t, (a_t−1, a_t)) over all possible actions a_t.

Let { , …, } be the approximation spaces for the Q-functions. We assume that Q_t(s_t, a_t) = 0 whenever z_t = ∅. In other words, if a failure occurred before the t-th-time-point, Q_t equals zero.

Note that by (3), the optimal t-stage Q-function $Q_t^{\ast }({\mathit{s}}_t,{\mathit{a}}_t)$ equals the conditional expectation of $R_t+{max}_{a_{t+1}}{Q}_{t+1}^{\ast }({\mathit{S}}_{t+1},({\mathit{A}}_t,a_{t+1}))$ given (s_t, a_t). Thus

$$Q_t^{\ast }=\underset{Q_t}{\text{argmin}}E\left[{\left(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{\mathit{t}+\mathbf{1}},({\mathit{A}}_{\mathit{t}},a_{t+1}))-Q_t({\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}})\right)}^2\right].$$

Ideally, we could compute the functions Q̂_t using backward recursion in the following way:

$${\widehat{Q}}_t=\underset{Q_t}{\text{argmin}}{\mathbb{E}}_n\left[{\left(R_t+\underset{a_{t+1}}{max}{\widehat{Q}}_{t+1}({\mathit{S}}_{\mathit{t}+\mathbf{1}},({\mathit{A}}_{\mathit{t}},a_{t+1}))-Q_t({\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}})\right)}^2\right],$$

where is the empirical expectation. The problem is that R_t may be censored and thus unknown.

Note that $E[{\delta }_t\mid {\sum }_{i=1}^t{R}_i]=P(C\ge {\sum }_{i=1}^t{R}_i)=S_C({\sum }_{i=1}^t{R}_i)$ and thus

$$E\left[\frac{{\delta }_t}{S_C({\sum }_{i=1}^t{R}_i)}|{\mathit{S}}_t,{\mathit{A}}_t,R_t\right]=1$$

since S_t includes the information regarding R₁, …, R_t−1 and C is independent of the covariates and actions.

Thus, for every function Q_t ∈ ,

$$\begin{array}{l}E\left[{\left(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{\mathit{t}+\mathbf{1}},({\mathit{A}}_{\mathit{t}},a_{t+1}))-Q_t({\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}})\right)}^2\right]=E\left[{\left(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{\mathit{t}+\mathbf{1}},({\mathit{A}}_{\mathit{t}},a_{t+1}))-Q_t({\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}})\right)}^{2}E\left[\frac{{\delta }_t}{S_C({\sum }_{i=1}^t{R}_i)}|{\mathit{S}}_t,{\mathit{A}}_t,R_t\right]\right]\\ =E\left[E\left[{\left(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{\mathit{t}+\mathbf{1}},({\mathit{A}}_{\mathit{t}},a_{t+1}))-Q_t({\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}})\right)}^2\frac{{\delta }_t}{S_C({\sum }_{i=1}^t{R}_i)}|{\mathit{S}}_t,{\mathit{A}}_t,R_t\right]\right]\\ =E\left[{\left(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{\mathit{t}+\mathbf{1}},({\mathit{A}}_{\mathit{t}},a_{t+1}))-Q_t({\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}})\right)}^2\frac{{\delta }_t}{S_C({\sum }_{i=1}^t{R}_i)}\right].\end{array}$$ (12)

Since $Q_t^{\ast }$ is the minimizer of the first expression in the above sequence of equalities, it also minimize the last expression. Thus, we suggest to choose Q̂_t recursively as follows:

$$\underset{Q_t\in {\mathcal{Q}}_t}{\text{argmin}}{\mathbb{E}}_n\left[{\left(R_t+\underset{a_{t+1}}{max}{\widehat{Q}}_{t+1}({\mathit{S}}_{\mathit{t}+\mathbf{1}},({\mathit{A}}_{\mathit{t}},a_{t+1}))-Q_t({\mathit{S}}_{\mathit{t}},{\mathit{A}}_{\mathit{t}})\right)}^2\frac{{\delta }_t}{{\widehat{S}}_C({\sum }_{i=1}^t{R}_i)}\right],$$ (13)

where we define Q̂_T+1 ≡ 0, and Ŝ_C is the Kaplan-Meier estimator of the survival function of the censoring variable S_C. Note that by Remark 3.1, the Kaplan-Meier estimator at x needs to estimate P(C ≥ x) rather than P(C > x). This can be done by taking left a continuous version of the Kaplan-Meier estimator that interchanges the roles of failure and censoring events for estimation (see Satten and Datta, 2001).

We define the policies π̂_t using the approximated Q-functions Q̂_t as follows:

$${\widehat{\pi }}_t({\mathit{s}}_t,{\mathit{a}}_{t-1})=\underset{a_t}{\text{argmax}}{\widehat{Q}}_t({\mathit{s}}_t,({\mathit{a}}_{t-1},a_t)).$$

6. Theoretical Results

Let { , …, } be the approximation spaces for the minimization problems (13). Note that we do not assume that the problem is Markovian, but, instead, we assume that each Q_t is a function of all the history up to and including stage t. Hence the spaces can be different over t.

We assume that the absolute values of the functions in the spaces { }_t are bounded by some constant M. Moreover, we need to bound the complexity of the spaces { }_t. We choose to use uniform entropy as the complexity measure (see van der Vaart and Wellner, 1996). This enables us to obtain exponential bounds on the difference between the true and empirical expectation of the loss function that involves a random component, namely, the Kaplan-Meier estimator, as in (13) (see Lemma A.6 ). This is different from Murphy (2005b) who uses the covering number as a measure of complexity (Anthony and Bartlett, 1999, pg 148) for the squared error loss function.

For every ε > 0 and measure P, we denote the covering number of by N(ε, , L₂(P )), where N (ε, , L₂(P )) is the minimal number of closed L₂(P)-balls of radius ε required to cover . The uniform covering number of is defined as sup_P N (εM, , L₂(P )) where the supremum is taken over all finitely discrete probability measures P on . The log of the uniform covering number is called the uniform entropy (van der Vaart and Wellner, 1996, page 84). We assume the following uniform entropy bound for the spaces { }:

$$\underset{t=\{1,\dots ,T\}}{max}\underset{P}{sup}logN(\epsilon M,{\mathcal{Q}}_t,L_2(P))<D{\left(\frac{1}{\epsilon }\right)}^W,$$ (14)

for all 0 < ε ≤ 1 and some constants 0 < W < 2 and D < ∞, where the supremum is taken over all finitely discrete probability measures, and M is the uniform bound defined above.

In the following, we prove a finite sample bound on the difference between the expected truncated survival times of an optimal policy and the policy π̂ obtained by the algorithm. As a corollary we obtain that the difference converges to zero under certain conditions.

The proof of the theorem consists of the following steps. First we use Lemma 4.1 to map the original problem to the corresponding auxiliary one. Second, for the auxiliary problem, we adapt arguments given in Murphy (2005b) to bound the difference between the expected value of the learned policy and the expected value of the optimal policy using error terms that involve expectations of both the learned and optimal Q-functions. Third, we bound these error terms by decomposing them to terms that arise due to the difference between the empirical and true expectation, terms that arise due the differences between the estimated and true censoring distribution, and terms that related to the empirical difference between the estimated and optimal Q-function. Fourth, and finally, we obtain a finite sample bound which depends on the complexity of the spaces { }, the deviation of the Kaplan-Meier estimator from the censoring distribution, and the size of the empirical errors in (13).

Theorem 6.1

Let { , …, } be the approximation spaces for the Q-functions. Assume that the uniform entropy bound (14) holds. Assume that n trajectories are sampled according to P₀. Let π̂ be defined by (4).

Then for any 0 < η < 1, we have with probability at least 1 − η, over the random sample of trajectories,

$$\begin{array}{l}\underset{\pi \in \mathrm{\Pi }}{sup}{E}_{0,\pi }\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau \right]-E_{0,\widehat{\pi }}\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau \right]\\ \le 16\epsilon +\\ \sum _{t=1}^T{L}^{t/2}\sum _{j=t}^T{\left(2L^j{4}^{j-t}{\mathbb{E}}_n{\left[\frac{{\delta }_t}{{\widehat{S}}_C({\sum }_{i=1}^t{R}_i)}\left(F({\widehat{Q}}_t,{\widehat{Q}}_{t+1})-F(Q_t^{\ast },{\widehat{Q}}_{t+1})\right)\right]}_{+}\right)}^{1/2}\end{array}$$ (15)

for all n that satisfies

$$max\left\{\frac{5T}{2}exp\{-{nC}_1{\epsilon }^4+\sqrt{n}{C}_2{\epsilon }^2\},{TC}_{3}exp\{-2n{\epsilon }^4+C_4\sqrt{n}{\epsilon }^{2(U+{\alpha }_o)}\right\}<\frac{\eta }{2},$$

where

$$\begin{array}{l}F(Q_t,Q_{t+1})={(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-Q_t)}^2,\\ C_1=2{(1-G_{min})}^2{M}_1^{-2}{K}_{min}^4{(4L)}^{-2(T+1)},\\ C_2=C_o(1-G_{min})M_1^{-1}{K}_{min}^2{(4L)}^{-(T+1)},\\ C_3=C_{a}exp\{{(4L)}^{-(T+1)}\},\\ C_4=C_b{(4L)}^{(T+1)/2},\end{array}$$

and where M₁ = (2M + τ)², C_o is the constant that appears in Bitouzé et al. (1999, Eq. 1), C_a, C_b, and U are the constants that appear in Lemma A.6, and for some α_o small enough such that U + α_o < 2.

Before we begin the proof of Theorem 6.1, we note that the bound (15) cannot be used in practice to perform structural risk minimization (see, for example, Vapnik, 1999) for two reasons. First, the bound itself is too loose (see also Murphy, 2005b, Theorem 1, Remark 4). Second, the constants, such as C_a and C_b, are not given, and are model dependent. Interestingly, a bound on C_o was established recently by Wellner (2007). However, this bound is large and simulations suggest that it is not tight. The bound (15) can, however, be used to derive asymptotic rates (Steinwart and Chirstmann, 2008, Chapter 6). Moreover, when the functions $Q_t^{\ast }$ are in , we obtain universal consistency, as stated in the following corollary:

Corollary 6.2

Assume that the conditions of Theorem 6.1 hold. Assume also that for every t, $Q_t^{\ast }\in {\mathcal{Q}}_t$. Then

$$\underset{\pi \in \mathrm{\Pi }}{sup}{E}_{0,\pi }\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau \right]-E_{0,\widehat{\pi }}\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau \right]\stackrel{\text{a}.\text{s}.}{\to }0.$$

Proof of Corollary 6.2

Note that for every t, Q̂_t is the minimizer of

$${\mathbb{E}}_n\left[\frac{{\delta }_t}{{\widehat{S}}_C({\sum }_{i=1}^t{R}_i)}F(Q_t,{\widehat{Q}}_{t+1})\right].$$

Hence, the second expression in the right hand side of (15) equals zero, and the result follows.

Proof of Theorem 6.1

By Lemma 4.1,

$$\underset{\pi \in \mathrm{\Pi }}{sup}{E}_{0,\pi }\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau \right]-E_{0,\widehat{\pi }}\left[\left({\sum }_{t=1}^{\overline{T}}{R}_t\right)\wedge \tau \right]=E[V^{\ast }(S_1)-V_{\widehat{\pi }}(S_1)],$$

where the expectation on the right hand side of the equality is with respect to the modified distribution P.

By Lemma 2 of Murphy (2005b) and Remark 2 that follows, for every state s_o ∈ ,

$$V^{\ast }(s_o)-V_{\widehat{\pi }}(s_o)\le \sum _{t=1}^{T}2L^{t/2}\sqrt{E\left[{\left({\widehat{Q}}_t({\mathit{S}}_t,{\mathit{A}}_t)-Q_t^{\ast }({\mathit{S}}_t,{\mathit{A}}_t)\right)}^2|S_1=s_o\right]}.$$

Applying Jensen’s inequality, we obtain

$$E[V^{\ast }(S_1)-V_{\widehat{\pi }}(S_1)]\le \sum _{t=1}^{T}2L^{t/2}\sqrt{E\left[{\left({\widehat{Q}}_t({\mathit{S}}_t,{\mathit{A}}_t)-Q_t^{\ast }({\mathit{S}}_t,{\mathit{A}}_t)\right)}^2\right]}.$$ (16)

We wish to obtain a bound on the expression $E\left[{\left({\widehat{Q}}_t({\mathit{S}}_t,{\mathit{A}}_t)-Q_t^{\ast }({\mathit{S}}_t,{\mathit{A}}_t)\right)}^2\right]$ using the expressions ${\mathit{Err}}_{{\widehat{Q}}_{t+1}}({\widehat{Q}}_t)-{\mathit{Err}}_{{\widehat{Q}}_{t+1}}(Q_t^{\ast })$, where

$${\mathit{Err}}_{Q_{t+1}}(Q_t)=E\left[{\left(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-Q_t({\mathit{S}}_t,{\mathit{A}}_t)\right)}^2\right],$$

for any pair of function Q_t and Q_t+1. To obtain this bound we follow the line of arguments that leads to the bound in Eq. 13 in the proof of Theorem 1 of Murphy (2005b). The bound (19) obtained here is tighter since only the special case of $Q_t^{\ast }$ in the second Err function is considered. To simplify the following expressions, we write Q_t instead of Q_t(S_t, A_t) whenever no confusion could occur.

For each t,

$$\begin{array}{l}{\mathit{Err}}_{{\widehat{Q}}_{t+1}}({\widehat{Q}}_t)-{\mathit{Err}}_{{\widehat{Q}}_{t+1}}(Q_t^{\ast })\\ =E[{\widehat{Q}}_t^2]-E[{(Q_t^{\ast })}^2]+2E[(R_t+\underset{a_{t+1}}{max}{\widehat{Q}}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1}))(Q_t^{\ast }-{\widehat{Q}}_t)]\\ =E[{\widehat{Q}}_t^2]-E[{(Q_t^{\ast })}^2]+2E[(Q_t^{\ast }-{\widehat{Q}}_t)E[(R_t-\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1}))\mid {\mathit{S}}_t,{\mathit{A}}_t]]\\ +2E[(\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-\underset{a_{t+1}}{max}{\widehat{Q}}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1}))(Q_t^{\ast }-{\widehat{Q}}_t)]\\ =E[{\widehat{Q}}_t^2]-E[{(Q_t^{\ast })}^2]+2E[{(Q_t^{\ast })}^2]-2E[{\widehat{Q}}_t{Q}_t^{\ast }]\\ +2E[(\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-\underset{a_{t+1}}{max}{\widehat{Q}}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1}))(Q_t^{\ast }-{\widehat{Q}}_t)]\\ =E{[({\widehat{Q}}_t-Q_t^{\ast })}^2]\\ +2E[(\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-\underset{a_{t+1}}{max}{\widehat{Q}}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1}))(Q_t^{\ast }-{\widehat{Q}}_t)],\end{array}$$ (17)

where the second to the last equality follows since

$$Q_t^{\ast }({\mathit{s}}_t,{\mathit{a}}_t)=E[(R_t-\underset{a_{t+1}}{max}{\widehat{Q}}_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1}))\mid {\mathit{S}}_t={\mathit{s}}_t,{\mathit{A}}_t={\mathit{a}}_t].$$

Using the Cauchy-Schwarz inequality for the second expression of (17), we obtain

$$\begin{array}{l}{\mathit{Err}}_{{\widehat{Q}}_{t+1}}({\widehat{Q}}_t)-{\mathit{Err}}_{{\widehat{Q}}_{t+1}}(Q_t^{\ast })\\ \ge E[{({\widehat{Q}}_t-Q_t^{\ast })}^2]\\ -2E{[{(\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-\underset{a_{t+1}}{max}{\widehat{Q}}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1}))}^2]}^{1/2}E{[{(Q_t^{\ast }-{\widehat{Q}}_t)}^2]}^1\end{array}$$

Note that

$$\begin{array}{l}E[{\left(\underset{a_{t+1}}{max}{Q}_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-\underset{a_{t+1}}{max}{\widehat{Q}}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})\right)}^2]\le E\left[\underset{a_{t+1}}{max}{\left(Q_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-{\widehat{Q}}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})\right)}^2\right]\\ \le E\left[L\sum _{a\in \mathcal{A}}{\left(Q_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_t,a)-{\widehat{Q}}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a)\right)}^2{p}_t(a\mid {\mathit{S}}_{t+1},{\mathit{A}}_t)\right]\\ =LE\left[{\left(Q_{t+1}^{\ast }({\mathit{S}}_{t+1},{\mathit{A}}_{t+1})-{\widehat{Q}}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_{t+1})\right)}^2\right],\end{array}$$ (18)

where the first inequality follows since (max_a h(a) −max_a h′(a))² ≤ max_a(h(a) −h′(a))² and where L is the constant that appears in the definition of the exploration policy p (see Section 3).

Using inequality (18) and the fact that $xy\le \frac{1}{2}(x^2+y^2)$, we obtain

$$\begin{array}{l}{\mathit{Err}}_{{\widehat{Q}}_{t+1}}({\widehat{Q}}_t)-{\mathit{Err}}_{{\widehat{Q}}_{t+1}}(Q_t^{\ast })\ge E[{({\widehat{Q}}_t-Q_t^{\ast })}^2]-E{[4L{(Q_{t+1}^{\ast }-{\widehat{Q}}_{t+1})}^2]}^{1/2}E{[{(Q_t^{\ast }-{\widehat{Q}}_t)}^2]}^{1/2}\\ \ge \frac{1}{2}E[{({\widehat{Q}}_t-Q_t^{\ast })}^2]-2LE[{(Q_{t+1}^{\ast }-{\widehat{Q}}_{t+1})}^2].\end{array}$$

Hence

$$E[{({\widehat{Q}}_t-Q_t^{\ast })}^2]\le 2\left({\mathit{Err}}_{{\widehat{Q}}_{t+1}}({\widehat{Q}}_t)-{\mathit{Err}}_{{\widehat{Q}}_{t+1}}(Q_t^{\ast })\right)+4LE[{(Q_{t+1}^{\ast }-{\widehat{Q}}_{t+1})}^2].$$

Using the fact that ${\widehat{Q}}_{T+1}=Q_{T+1}^{\ast }=0$ we obtain

$$E[{({\widehat{Q}}_t-Q_t^{\ast })}^2]\le 2\sum _{j=t}^T{(4L)}^{j-t}\left({\mathit{Err}}_{{\widehat{Q}}_{j+1}}({\widehat{Q}}_j)-{\mathit{Err}}_{{\widehat{Q}}_{j+1}}(Q_j^{\ast })\right).$$ (19)

We are now ready to bound the expressions ${\mathit{Err}}_{{\widehat{Q}}_{j+1}}({\widehat{Q}}_j)-{\mathit{Err}}_{{\widehat{Q}}_{j+1}}(Q_j^{\ast })$. For any $Q_t\in {\mathcal{Q}}_t\cup Q_t^{\ast }$, Q_t+1 ∈ , and censoring survival function K: [0, τ] ↦ [K_min, 1], where K_min > 0, define

$$\begin{array}{l}\mathcal{E}(Q_t,Q_{t+1},K)=E\left[\frac{{\delta }_t}{K({\sum }_{i=1}^t{R}_i)}{(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-Q_t)}^2\right],\\ {\mathcal{E}}_n(Q_t,Q_{t+1},K)={\mathbb{E}}_n\left[\frac{{\delta }_t}{K({\sum }_{i=1}^t{R}_i)}{(R_t+\underset{a_{t+1}}{max}{Q}_{t+1}({\mathit{S}}_{t+1},{\mathit{A}}_t,a_{t+1})-Q_t)}^2\right].\end{array}$$ (20)

Note that similarly to (12) we have Err_Q̂t+1(Q_t) = ε(Q_t, Q̂_t+1, S_C), where S_C is the censoring survival function.

Using this notation we have

$$\begin{array}{l}{\mathit{Err}}_{{\widehat{Q}}_{t+1}}({\widehat{Q}}_t)-{\mathit{Err}}_{{\widehat{Q}}_{t+1}}(Q_t^{\ast })=\mathcal{E}({\widehat{Q}}_t,{\widehat{Q}}_{t+1},S_C)-\mathcal{E}(Q_t^{\ast },{\widehat{Q}}_{t+1},S_C)\\ \le \mid \mathcal{E}({\widehat{Q}}_t,{\widehat{Q}}_{t+1},S_C)-\mathcal{E}({\widehat{Q}}_t,{\widehat{Q}}_{t+1},{\widehat{S}}_C)\mid \\ +\mid \mathcal{E}({\widehat{Q}}_t,{\widehat{Q}}_{t+1},{\widehat{S}}_C)-{\mathcal{E}}_n({\widehat{Q}}_t,{\widehat{Q}}_{t+1},{\widehat{S}}_C)\mid \\ +({\mathcal{E}}_n({\widehat{Q}}_t,{\widehat{Q}}_{t+1},{\widehat{S}}_C)-{\mathcal{E}}_n(Q_t^{\ast },{\widehat{Q}}_{t+1},{\widehat{S}}_C))+\\ \mid {\mathcal{E}}_n(Q_t^{\ast },{\widehat{Q}}_{t+1},{\widehat{S}}_C)-\mathcal{E}(Q_t^{\ast },{\widehat{Q}}_{t+1},{\widehat{S}}_C)\mid \\ +\mid \mathcal{E}(Q_t^{\ast },{\widehat{Q}}_{t+1},{\widehat{S}}_C)-\mathcal{E}(Q_t^{\ast },{\widehat{Q}}_{t+1},S_C)\mid ,\end{array}$$

where Ŝ_C is the Kaplan-Meier estimator of S_C, and (a)₊ = max{a, 0}. Hence

$${\mathit{Err}}_{{\widehat{Q}}_{t+1}}({\widehat{Q}}_t)-{\mathit{Err}}_{{\widehat{Q}}_{t+1}}(Q_t^{\ast })\le 2\underset{\{Q_t,Q_{t+1}\}}{sup}\mid \mathcal{E}(Q_t,Q_{t+1},S_C)-\mathcal{E}(Q_t,Q_{t+1},{\widehat{S}}_C)\mid$$ (21)

$$+2\underset{\{Q_t,Q_{t+1},K\}}{sup}\mid \mathcal{E}(Q_t,Q_{t+1},K)-{\mathcal{E}}_n(Q_t,Q_{t+1},K)\mid$$ (22)

$$+({\mathcal{E}}_n({\widehat{Q}}_t,{\widehat{Q}}_{t+1},{\widehat{S}}_C)-{\mathcal{E}}_n(Q_t^{\ast },{\widehat{Q}}_{t+1},{\widehat{S}}_C))+.$$ (23)

Combining (19) and (21), and substituting in (16), we have

$$\begin{array}{l}E[V^{\ast }(S_1)-V_{\widehat{\pi }}(S_1)]\le 2\sum _{t=1}^T{L}^{t/2}\sum _{j=t}^T\sqrt{2{(4L)}^{j-t}({\mathit{Err}}_{Q_{t+1}}(Q_t)-{\mathit{Err}}_{Q_{t+1}}(Q_t^{\ast }))}\\ \le 8{(4L)}^{(T+1)/2}\sqrt{\underset{t}{max}\underset{\{Q_t,Q_{t+1}\}}{sup}\mid \mathcal{E}(Q_t,Q_{t+1},S_C)-\mathcal{E}(Q_t,Q_{t+1},{\widehat{S}}_C)\mid }\end{array}$$ (24)

$$\begin{array}{l}+8{(4L)}^{(T+1)/2}\sqrt{\underset{t}{max}\underset{\{Q_t,Q_{t+1},K\}}{sup}\mid \mathcal{E}(Q_t,Q_{t+1},K)-{\mathcal{E}}_n(Q_t,Q_{t+1},K)\mid }\\ +2\sum _{t=1}^T{L}^{t/2}\sum _{j=t}^T{2}^{j-t}{L}^{j/2}\sqrt{2{({\mathcal{E}}_n({\widehat{Q}}_t,{\widehat{Q}}_{t+1},{\widehat{S}}_C)-{\mathcal{E}}_n(Q_t^{\ast },{\widehat{Q}}_{t+1},{\widehat{S}}_C))}_{+}},\end{array}$$ (25)

where we used the fact that ${\sum }_{t=1}^T{L}^{t/2}{\sum }_{j=t}^T{(4L)}^{(j-t)/2}\le 2{(4L)}^{(T+1)/2}$ for L ≥ 2 and the fact that $\sqrt{x+y}\le \sqrt{x}+\sqrt{y}$.

In the following, we replace the bounds in (24) and (25)) with exponential bounds. We start with (24). Note that (R_t + max_at+1 Q_t+1(S_t+1, A_t, a_t+1) − Q_t)² ≤ M₁ = (2M + τ)² for all Q_t, Q_t+1. Hence,

$$\underset{\{Q_t,Q_{t+1}\}}{sup}\left|\mathcal{E}(Q_t,Q_{T+1},S_C)-\mathcal{E}(Q_t,Q_{T+1},{\widehat{S}}_C)\right|\le M_1{K}_{min}^{-2}E[\mid S_C-{\widehat{S}}_C\mid ],$$

, and thus

$$\begin{array}{l}P\left({(4L)}^{T/2+2}\sqrt{\underset{t}{max}\underset{\{Q_t,Q_{t+1}\}}{sup}\mid \mathcal{E}(Q_t,Q_{T+1},S_C)-\mathcal{E}(Q_t,Q_{T+1},{\widehat{S}}_C)}>\epsilon \right)\\ \le \sum _{t=1}^{T}P\left({(4L)}^{T/2+2}\sqrt{\underset{\{Q_t,Q_{t+1}\}}{sup}\mid \mathcal{E}(Q_t,Q_{T+1},S_C)-\mathcal{E}(Q_t,Q_{T+1},{\widehat{S}}_C)}>\epsilon \right)\\ \le TP\left({(4L)}^{T/2+2}\sqrt{M_1{K}_{min}^{-2}{\left|\right|S-{\widehat{S}}_C\left|\right|}_{\infty }}>\epsilon \right),\end{array}$$ (26)

where the first equality follows from the fact that

$$P(\underset{t\in \{1,\dots ,T\}}{max}{X}_t>c)\le \sum _{t=1}^{T}P(X_t>c).$$ (27)

Using a Dvoretzky-Kiefer-Wolfowitz-type inequality for the Kaplan-Meier estimator (Bitouzé et al., 1999, Theorem 2), we have

$$P({\left|\right|S_C-{\widehat{S}}_C\left|\right|}_{\infty }>{\epsilon }^{\prime })<\frac{5}{2}exp\{-2n{(1-G_{min})}^2{({\epsilon }^{\prime })}^2+C_o\sqrt{n}(1-G_{min}){\epsilon }^{\prime }\},$$ (28)

where C_o is some universal constant and G_min is a lower bound on the survival function at τ (see Section 3).

Write $\epsilon ={(4L)}^{(T+1)/2}\sqrt{M_1{K}_{min}^{-2}{\epsilon }^{\prime }}$, and thus ${\epsilon }^{\prime }=M_1^{-1}{K}_{min}^2{\epsilon }^2{(4L)}^{-(T+1)}$. Note that $8{(4L)}^{T/2+2}\sqrt{M_1{K}_{min}^{-2}{\left|\right|S_C-{\widehat{S}}_C\left|\right|}_{\infty }}>8\epsilon \text{iff}{\left|\right|S_C-{\widehat{S}}_C\left|\right|}_{\infty }>{\epsilon }^{\prime }$ . Applying the inequality (28) to the right hand-side of (26) and substituting for ε, we obtain

$$\begin{array}{l}P\left(8{(4L)}^{(T+1)/2}\sqrt{\underset{t}{sup}\underset{\{Q_t,Q_{t+1}\}}{sup}\mid \mathcal{E}(Q_t,Q_{T+1},S_C)-\mathcal{E}(Q_t,Q_{T+1},{\widehat{S}}_C)}>8\epsilon \right)\\ \le \frac{5T}{2}exp\{-2n{(1-G_{min})}^2{M}_1^{-2}{K}_{min}^4{\epsilon }^4{(4L)}^{-2(T+1)}\\ +C_o\sqrt{n}(1-G_{min})M_1^{-1}{K}_{min}^2{\epsilon }^2{(4L)}^{-(T+1)}\}\\ \equiv \frac{5T}{2}exp\{-{nC}_1{\epsilon }^4+\sqrt{n}{C}_2{\epsilon }^2\},\end{array}$$ (29)

where $C_1=2{(1-G_{min})}^2{M}_1^{-2}{K}_{min}^4{(4L)}^{-2(T+1)}$ and $C_2=C_o(1-G_{min})M_1^{-1}{K}_{min}^2{(4L)}^{-(T+1)}$.

We now find an exponential bound for (25). We follow the same line of arguments, replacing the Dvoretzky-Kiefer-Wolfowitz type inequality used in the previous proof with the uniform entropy bound. Recall that by assumption, the uniform entropy bound (14) holds for the spaces _t and thus also for the spaces ${\mathcal{Q}}_t\cup Q_t^{\ast }$. Hence, by Lemma A.6, and (27), for W′ = max{W, 1} and for all α > 0, we have

$$\begin{array}{l}P\left(8{(4L)}^{(T+1)/2}\sqrt{\underset{t}{max}\underset{\{Q_t,Q_{t+1},K\}}{sup}\mid \mathcal{E}(Q_t,Q_{t+1},K)-{\mathcal{E}}_n(Q_t,Q_{t+1},K)\mid }>8\epsilon \right)\\ \le {TC}_{a}exp\{C_b\sqrt{n}{(4L)}^{-(T+1)/2}{\epsilon }^{2(U+\alpha )}-2n{(4L)}^{-(T+1)}{\epsilon }^4\}\\ \equiv {TC}_{3}exp\{C_4\sqrt{n}{\epsilon }^{2(U+\alpha )}-2n{\epsilon }^4\},\end{array}$$ (30)

where C₃ = C_a exp{(4L)^−(T+1)}, C₄ = C_b(4L)^(T+1)/2, and U = W′(6 − W′)/(2 + W′).

Take n large enough such that the right hand side of (29) and (30) are less than η/2 and substitute in (24) and (25), respectively, and the result of the theorem follows.

7. Simulation Study

We simulate a randomized clinical trial with flexible number of stages to examine the performance of the proposed censored Q-learning algorithm. We compare the estimated individualized treatment policy to various possible fixed treatments. We also compare the given expected survival times of different censoring levels. Finally, we test the effect of ignoring the censoring.

This section is organized as follows. We first describe the setting of the simulated clinical trial (Section 7.1). We then describe the implementation of the simulation (Section 7.2). The simulation results appear in Section 7.3.

7.1. Simulated Clinical Trial

We consider the following hypothetical cancer trial. The duration of the trial is 3 years. The state of each patient at each time-point u ∈ [0, 3] includes the tumor size (0 ≤ T(u) ≤ 1), and the wellness (0.25 ≤ W(u) ≤ 1). The time-point u_o such that W(u_o) < 0.25 is considered the failure time. We define the critical tumor size to be 1. At time u_i such that T(u_i) = 1, we begin a treatment. We call the duration [u_i, u_i+1] the i-th stage. Note that different patients may have different numbers of stages.

At each time-point u_i, we consider two optional treatments: a more aggressive treatment (A), and a less aggressive treatment (B). The immediate effects of treatment A are

$$\begin{array}{l}W(u_i^{+}\mid A)=W(u_i)-0.5,\\ T(u_i^{+}\mid A)=T(u_i)/(10W(u_i)),\end{array}$$ (31)

i.e., the wellness at time u_i after treatment A (denoted by $W(u_i^{+}\mid A)$) decreases by 0.5 wellness units. The tumor size at time u_i after treatment A (denoted by $T(u_i^{+}\mid A)$) decreases by a factor of 1=(10W (u_i)) which reflects a greater decrease of tumor size for a larger wellness value. Similarly, the immediate effects of the less aggressive treatment B are

$$\begin{array}{l}W(u_i^{+}\mid B)=W(u_i)-0.25,\\ T(u_i^{+}\mid B)=T(u_i)/(4W(u_i)),\end{array}$$ (32)

which, in comparison to the treatment A, has lower effect on the tumor size but also lower decrease of wellness. The wellness and tumor size at time u_i < u ≤ u_i+1 follows the dynamics

$$\begin{array}{l}W(u)=W(u_i^{+})+(1-W(u_i^{+}))(1-2^{-(u-u_i)/2}),\\ T(u)=T(u_i^{+})+4T(u_i^{+})(u-u_i)/3.\end{array}$$ (33)

The stage that begins at time-point u_i ends when either T(u_i+1) = 1 for some u_i < u_i+1 < 3 or when a failure event occurs or at the end of the trial when u = 3. During this stage, we model the survival function of the patient as an exponential distribution with mean $3(W(u_i^{+})+2)/20M(u_i^{+})$.

The trajectories are constructed as follows. We assume that patients are recruited to the trial when their tumor size reaches the critical size, i.e., for all patients T(0) = 1, and hence u₁ = 0 is the beginning of the first stage. The wellness at the beginning of the first stage, W(0), is uniformly distributed on the segment [0.5, 1]. With equal probability, a treatment a₁ ∈ {A, B} is chosen. If no failure event occurs during the first stage, the first stage ends when either T(u₂) = 1 for some 0 = u₁ < u₂ < 3 or at the end of the trial. If the first stage ends before the end of the trial, then with equal probability another treatment a₂ ∈ {A, B} is chosen. The trial continues in the same way until either a failure time occurs or the trial ends. We note that the actual number of stages for each patient is a random function of the initial state and the treatments chosen during the trial. Due to the choices of model parameters, the number of stages in the above dynamics is at least one and not more than three.

For each trajectory, a censoring variable C is uniformly drawn from the segment [0, c] for some constant c > 3, where the choice of the constant c determines the expected percentage of censoring. When an event is censored, the trajectory (i.e., the states and treatments) up to the point of censoring and the censoring time are given.

7.2. Simulation Implementation

The Q-learning algorithm presented in Section 5 was implemented in the Matlab environment. For the implementation we used the Spider library for Matlab¹. The Matlab code, as well as the data sets, are available online (see Supplement A).

The algorithm is implemented as follows. The input for the algorithm is a set of trajectories obtained according to the dynamics described in Section 7.1. First, the Kaplan-Meier estimator for the survival function of the censoring variable is computed from the given trajectories. Then, we set Q̂₄ ≡ 0 and compute Q̂_i, i = 3, 2, 1 backwardly, as the minimizer of (13) over all the functions Q_i(s_i, a_i) which are linear in the first variable. The policy π̂ is computed from the functions {Q̂₁, Q̂₂, Q̂₃} using (4).

We tested the policy π̂ = (π̂₁, π̂₂, π̂₃) by constructing 1000 new trajectories, in which the choice of treatment at each stage is according to π̂. 1000 initial wellness values were drawn uniformly from the segment [0.5, 1]. For each wellness value, a treatment was chosen from the set {A, B}, according to the policy π̂₁. The immediate effect of the treatment was computed according to (31)–(32). A failure time was drawn from the exponential distribution with mean as described in the previous section; denote this time by f₁. The time that the tumor reached the critical size was computed according to the dynamics (33), and we denote this time by u₂. If both f₁ and u₂ are greater than 3 (the end of the trial) then the trajectory was ended after the first stage and the survival time for this patient was given as 3. Otherwise, if f₁ ≤ u₂, the trajectory is ended after the first stage and the survival time for this patient was given as f₁. If u₂ < f₁, then at time u₂, a second treatment is chosen according to the policy π̂₂. The computation of the reminder of the trajectory is done similarly. The expected value of the policy π̂ is estimated by the mean of the survival time of all 1000 patients.

We compared the results of the algorithm to all fixed treatment sequences A₁A₂A₃, where A_i ∈ {A, B}. The expected values of the fixed treatment sequences were computed explicitly. We also compared the results to that of the optimal policy, which was also computed explicitly.

7.3. Simulation and Results

First, we would like to examine the influence of the sample size and censoring percentage on the algorithm’s performance. We simulated data sets of trajectories of sizes 40, 80, 120, …, 400. For each set of trajectories we considered four levels of censoring: no censoring, 10%-censoring, 20%-censoring, and 30%-censoring. Higher levels of (uniform) censoring were not considered since this requires drawing the censoring variable from a segment [0, c] for c < 3, which is in contrast to the assumption on the censoring variable (see the beginning of section 3). A policy π̂ was computed for each combination of data set size and censoring percentage. The policy π̂ was evaluated on a data set of size 1000, as described in Section 7.2. We repeated the simulation 400 times for each combination of data set size and censoring percentage. The mean values of the estimated mean survival time are presented in Figure 1. A comparison between the different fixed policies, polices obtained by the algorithm for different censoring levels, and the optimal policy appears in Figure 2. As can be seen from both figures, the individualized treatment polices obtained by the algorithm are better than any fixed policy. Moreover, as the number of observed trajectories increases, the expected survival time increases, for all censoring percentages.

Fig 1.

The solid black curve, dashed blue curve, dot-dashed red curve, and dotted green curve correspond to the expected survival time (in months) for different data set sizes with no censoring, 10% censoring, 20% censoring, and 30% censoring, respectively. The expected survival time was computed as the mean of 400 repetitions of the simulation. The black straight line, blue dashed straight line, and the dot-dashed red straight line correspond to the expected survival times of the optimal policy, the best fixed treatment policy, and the average of the fixed treatment policies, respectively.

Fig 2.

The eight light gray bars represent the expected survival times for different fixed treatments where A₁A₂A₃ indicates the policy that chooses A_i at the i-th stage. The four dark gray bars represent the expected survival times for policy π̂ obtained by the algorithm with no censoring, 10% censoring, 20% censoring, and 30% censoring. The white bar is the expected value of the optimal policy. The values of the fixed treatments and the optimal policy were computed analytically while the values of π̂ are the means of 400 repetitions of the simulation on 200 trajectories.

We also examined the influence of the sample size and censoring percentage on the distribution of estimated expected survival time. We simulated data sets of sizes 50, 100, 200, …, 3200 and we considered the four levels of censoring as before. As can be seen from Figure 3, the variance decreases when the sample size becomes larger. Also, the variance is smaller for smaller percentage of censoring, although the difference is modest.

Fig 3.

Distribution of expected survival time (in months) for different data set sizes, with no censoring, 10% censoring, 20% censoring, and 30% censoring. Each box plot is based on 400 repetitions of the simulation for each given data set size and censoring percentage.

Note that the maximum expected survival times obtained by the algorithm are a little bit above 17 months (see both Figures 1 and 2), while the value of the optimal policy is 17.85. The difference follows from the fact that the Q-functions estimated by the algorithm are linear while the optimal Q-function is not (see Figure 4). It is worth mentioning that even in the class of linear functions on which the optimization is done there are Q-functions that yield higher values. This fact is often referred to as the “mismatch” that follows from the fact that optimization of the value function is not performed explicitly, but rather through optimization of the Q-functions (see Tsitsiklis and van Roy, 1996; Murphy, 2005b, for more details).

Fig 4.

The Q-functions computed by the proposed algorithm for a size 200 trajectory set. The left panel presents both the optimal Q-function (solid red curve) and the estimated Q-function (dashed blue curve) for different wellness levels and when treatment A is chosen. Similarly, the middle panel shows both Q-functions when treatment B is chosen. The right panel shows the optimal value function (solid red curve) and the estimated value function (dashed blue curve).

Figure 5 shows the number of treatments that were needed for patients that followed the policy π̂ and did not have a failure event during the trial. As can be seen from this figure, patients with high initial wellness need only one treatment. On the other hand, patients with very low initial wellness value need three treatments.

Fig 5.

The number of required treatments for patients that follow the policy π̂, when no failure event occurs during the trial. The policy π̂ was estimated from 100 trajectories. The results were computed using a size 100, 000 testing set.

Finally, we checked the effect of ignoring the censoring on the expected survival time. We considered two ways of ignoring the censoring. First, we consider an algorithm that ignores the weights in the minimization problem (13). This is equivalent to deleting the last stage from each trajectory that was censored. We also consider an algorithm that deletes all censored trajectories. In the example presented in Figures 1–5, where uniform censoring takes place, there is a relatively moderate difference between the expected survival time for the proposed algorithm and the other two algorithms that ignore censoring. However, when the censoring variable follows the exponential distribution (leaving fewer observations with longer survival times) the bias from ignoring the censored trajectories is substantial, as can be seen in Figure 6.

Fig 6.

The solid blue curve, dashed black curve, and dot-dashed red curve correspond to the expected survival times (in months) for different data set sizes, for the proposed algorithm, the algorithm that ignores the weights, and the algorithm that deletes all censored trajectories, respectively. The censoring variable follows the exponential distribution with 50% censoring on average. The expected survival time was computed as the mean of 400 repetitions of the simulation.

8. Summary

We studied a framework for multistage-decision problems with flexible number of stages in which the rewards are survival times and are subject to censoring. We proposed a novel Q-learning algorithm adjusted for censoring. We derived the generalization error properties of the algorithm and demonstrated the algorithm performance using simulations.

The work as presented is applicable to real-world multi-stage decision problems with censoring. However, two main issues should be noted. First, we assumed that censoring is independent of observed trajectories. It would be useful to relax this assumption and allow censoring to depend on the covariates. Developing an algorithm that works under this relaxed assumption is a challenge. Second, we have used the inverse-probability-of-censoring weighting to correct the bias induced by censoring. When the percentage of censored trajectories is large, the algorithm may be inefficient. Finding a more efficient algorithm is also an open question.

APPENDIX A: SUPPLEMENTARY PROOFS

The main goal of this section is to provide an exponential bound on the difference between the empirical expectation (Q_t, Q_t+1, K) and the true expectation (Q_t, Q_t+1, K) as a function of the uniform entropy of the class of functions (see (20)). This result appears in Lemma A.6. Similar results for Glivenko–Cantelli classes, Donsker classes, and bounded uniform entropy integral (BUEI) classes can be found in van der Vaart and Wellner (1996) and Kosorok (2008).

Lemma A.1

Let , …, be k sets of functions. Assume that for every j ∈ {1, …, k}, sup_f∈ ||f||_∞ ≤ M_j. Let φ : ℝ^k ↦ ℝ satisfy

$${\mid \phi \circ f(x)-\phi \circ g(x)\mid }^2\le c^2\sum _{j=1}^k{(f_j(x)-g_j(x))}^2$$ (34)

for every f = (f₁, …, f_k), g = (g₁, …, g_k) ∈ × … × , where 0 < c < ∞. Let P be a finitely discrete probability measure. Define φ ∘ ( , …, ) = {φ(f₁, …, f_k) : (f₁, …, f_k) ∈ × … × }. Then

$$N(\epsilon c\sum _{j=1}^k{M}_j,\phi \circ ({\mathcal{F}}_1,\dots ,{\mathcal{F}}_k),L_2(P))\le \prod _{j=1}^{k}N(\epsilon M_j,{\mathcal{F}}_j,L_2(P)).$$ (35)

Proof

The proof is similar to the proof of Kosorok (2008, Lemma 9.13). Let f, g ∈ × … × satisfy ||f_j − g_j||_P,2 < εM_j for 1 ≤ j ≤ k. Note that

$${\left|\right|\phi \circ f-\phi \circ g\left|\right|}_{P,2}\le c\sqrt{{\sum }_{j=1}^k{\left|\right|f_j-g_j\left|\right|}_{P,2}^2}\le c\epsilon \sum _{j=1}^k{M}_j$$

which implies (35).

The following two corollaries are direct result of Lemma A.1:

Corollary A.2

Let = {K : K is monotone decreasing K : [0, τ] ↦ [K_min, 1]}. Define = {1/K : K ∈ }. Let P be a finitely discrete probability measure. Then

$$N(\epsilon K_{min}^{-1},{\mathcal{K}}^{-1},L_2(P))\le N(\epsilon ,\mathcal{K},L_2(P)).$$

Proof

Note that inequality (34) holds for k = 1 and $c=K_{min}^{-1}$, and the results follow from Lemma A.1.

Corollary A.3

Let = {Q(x, a) : x ∈ ℝ^p, a ∈ {1, …, k}, ||Q||_∞ ≤ M}. Define = {max_a Q(x, a) : Q ∈ }. Let P be a finitely discrete probability measure. Then

$$N(\epsilon kM,{\mathcal{Q}}^{max},L_2(P))\le N{(\epsilon M,\mathcal{Q},L_2(P))}^k.$$

Proof

Since (max_a h(a) − max_a h′(a))² ≤ max_a(h(a) − h′(a))², inequality (34) holds for c = 1. The results now follow from Lemma A.1.

We also need the following lemma and its corollary:

Lemma A.4

Let and F₂ be two function classes uniformly bounded in absolute value by M₁ and M₂, respectively. Define · = {f₁·f₂ : f_i ∈ }. Then

$$N(2\epsilon M_1{M}_2,{\mathcal{F}}_1·{\mathcal{F}}_2,L_2(P))\le N(\epsilon M_1,{\mathcal{F}}_1,L_2(P))·N(\epsilon M_2,{\mathcal{F}}_2,L_2(P)).$$

Proof

Let ||f_j −g_j||_P,2 ≤ εM_j where f_j, g_j ∈ , j = {1, 2}. Note that

$$\begin{array}{l}{\left|\right|f_1·f_2-g_1·g_2\left|\right|}_{P,2}\le {\left|\right|f_1(f_2-g_2)\left|\right|}_{P,2}+{\left|\right|g_2(f_1-g_1)\left|\right|}_{P,2}\\ \le M_1{\left|\right|f_2-g_2\left|\right|}_{P,2}+M_2{\left|\right|f_1-g_1\left|\right|}_{P,2}\le 2M_1{M}_2\epsilon .\end{array}$$

The result follows.

Corollary A.5

Let be a function class uniformly bounded in absolute value by M. Define = {g² : g ∈ }. Then

$$N(2\epsilon M^2,{\mathcal{G}}^2,L_2(P))\le N{(\epsilon M,\mathcal{G},L_2(P))}^2.$$

Proof

Apply Lemma A.4 with = = .

We use the previous results to prove the following lemma:

Lemma A.6

Let

$$\begin{array}{l}{\mathcal{Q}}_t\subset \{Q_t(x,a):x\in {ℝ}^{p_t},a\in \{1,\dots ,k\},{\left|\right|Q_t\left|\right|}_{\infty }\le M\},\\ \mathcal{K}=\{K:K\text{is}\text{monotone}\text{decreasing}K:[0,\tau ]\mapsto [K_{min},1]\},\\ ℛ=\left\{\frac{1}{K(t)}{\left(r+\underset{a}{max}{Q}_{t+1}(x,a)-Q_t(x,a)\right)}^2:r\in [0,\tau ],Q_t\in {\mathcal{Q}}_t,{\mathcal{Q}}_{t+1}\in {\mathcal{Q}}_{t+1},K\in \mathcal{K}\right\},\end{array}$$

where t ∈ 1, …, T and = {0}. Assume that the uniform entropy bound for each of the spaces (14) holds. Then

• 1There are constants D′ and W′ such that $logN(\epsilon ,ℛ,L_2(P))\le D^{\prime }{\left(\frac{1}{\epsilon }\right)}^{W^{\prime }}$, where W′ = max{W, 1}.
• 2
  For every α > 0 and t > 0,

  $$P^{\ast }(\underset{f\in ℛ}{sup}||Ef-{\mathbb{E}}_{n}f||>t)\le C_{a}exp\{C_b{\sqrt{nt}}^{U+\alpha }-2{nt}^2\},$$

  where U = W′(6−W′)/(2+W′), the constants C_a and C_b depend only on D′, W′ and α, and where P* is outer probability.

Proof

Let W′ = max{W, 1}. Note that uniform entropy bound (14) for the spaces holds also for W′. Note that by Corollary A.3, $logN(\epsilon M,{\mathcal{Q}}_t^{\text{max}},L_2(P))\le {Dk}^{W^{\prime }+1}{\left(\frac{1}{\epsilon }\right)}^{W^{\prime }}$. Since (x+y+z)² < 3(x²+y²+z²), we can apply Lemma A.1 to the class

$$\mathcal{G}=\{r+\underset{a}{max}{Q}_{t+1}(x,a)-Q_t(x,a):r\in [0,\tau ],Q_t\in {\mathcal{Q}}_t,Q_{t+1}\in {\mathcal{Q}}_{t+1}\},$$

with $c=\sqrt{3}$ and φ(x, y, z) = x+y+z to obtain $logN(\sqrt{3}\epsilon (2M+\tau ),\mathcal{G},L_2(P))\le (\tau +{Dk}^{W^{\prime }+1}+D){\epsilon }^{-W^{\prime }}$, where we used the fact that the segment [0, τ] can be covered by no more than τ/ε+1 balls of radius ε and that log(1+τ/ε) ≤ τ/ε. By Corollary A.5, we have $logN(2·3\epsilon (2M+\tau ),{\mathcal{G}}^2,L_2(P))\le 2(\tau +{Dk}^{W^{\prime }+1}+D){\left(\frac{1}{\epsilon }\right)}^{W^{\prime }}$ or, equivalently,

$$logN(\epsilon M_1,{\mathcal{G}}^2,L_2(P))\le D_1{\left(\frac{1}{\epsilon }\right)}^{W^{\prime }},$$

where M₁ = (2M +τ)² is a uniform bound for , and D₁ = 2(τ+Dk^W′+1+ D)6^−W′.

By Kosorok (2008, Lemma 9.11), log N(ε, , L₂(P)) ≤ D₂ε⁻¹ for some universal constant D₂ which is independent of the choice of probability measure P. By Corollary A.2,

$$logN(\epsilon K_{min}^{-1},{\mathcal{K}}^{-1},L_2(P))\le D_2\left(\frac{1}{\epsilon }\right).$$

Applying Lemma A.4 to = · , we obtain

$$logN(\epsilon K_{min}^{-1},M^{\prime },ℛ,L_2(P))\le (D_1+D_2){\left(\frac{1}{\epsilon }\right)}^{W^{\prime }}.$$

Since this inequality holds for every finitely discrete probability measures P, assertion 1 is proved. The second assertion follows from van der Vaart and Wellner (1996, Theorem 2.14.10).