Interactive Q-learning for Quantiles

Abstract

A dynamic treatment regime is a sequence of decision rules, each of which recommends treatment based on features of patient medical history such as past treatments and outcomes. Existing methods for estimating optimal dynamic treatment regimes from data optimize the mean of a response variable. However, the mean may not always be the most appropriate summary of performance. We derive estimators of decision rules for optimizing probabilities and quantiles computed with respect to the response distribution for two-stage, binary treatment settings. This enables estimation of dynamic treatment regimes that optimize the cumulative distribution function of the response at a prespecified point or a prespecified quantile of the response distribution such as the median. The proposed methods perform favorably in simulation experiments. We illustrate our approach with data from a sequentially randomized trial where the primary outcome is remission of depression symptoms.

1. Introduction

A dynamic treatment regime operationalizes clinical decision making as a series of decision rules that dictate treatment over time. These rules account for accrued patient medical history, including past treatments and outcomes. Each rule maps current patient characteristics to a recommended treatment, hence personalizing treatment. Typically, a dynamic treatment regime is estimated from data with the goal of optimizing the expected value of a clinical outcome, and the resulting regime is referred to as the estimated optimal regime.

Direct-search, also known as policy-search or value-search, is one approach to estimating an optimal dynamic treatment regime. Direct search estimators require a pre-specified class of dynamic treatment regimes and an estimator of the marginal mean outcome under any regime in the pre-specified class. The maximizer of the estimated marginal mean outcome over the class of regimes is taken as the estimator of the optimal dynamic treatment regime. Marginal structural models (MSMs) are one type of direct-search estimators (Robins, 2000; van der Laan et al., 2005; van der Laan, 2006; van der Laan and Petersen, 2007; Bembom and van der Laan, 2008; Robins et al., 2008; Orellana et al., 2010; Petersen et al., 2014). MSMs are best-suited to problems with a small class of potential regimes. MSMs may also be advantageous in practice because optimizing over a small class of pre-specified regimes provides a simpler, and often more interpretable, regime than other approaches. Another class of direct-search estimators casts the marginal mean outcome as a weighted missclassification rate and applies either discrete-optimization or classification algorithms to optimize a plugin estimator of the marginal mean outcome (Zhao et al., 2012; Zhang et al., 2012b,a, 2013; Zhao et al., 2015).

Regression-based or indirect estimators comprise a second class of estimators of an optimal dynamic treatment regime. Regression-based estimators require a model for some portion of the conditional distribution of the outcome given treatments and covariate information. Examples of regression-based estimators include Q-learning (Watkins, 1989; Watkins and Dayan, 1992; Murphy, 2005a), regularized Q-learning (Moodie and Richardson, 2010; Chakraborty et al., 2010; Song et al., 2015; Goldberg et al., 2013), Interactive Q-learning (Laber et al., 2014a), g-estimation in structural nested mean models (Robins, 2004), A-learning (Murphy, 2003), and regret-regression (Henderson et al., 2010). Nonparametric regression-based approaches often target the globally optimal regime rather than restricting attention to a small, pre-specified class. They can be also be useful in exploratory contexts to discover new treatment strategies for further evaluation in later trials.

Direct-search and regression-based estimators have been extended to handle survival outcomes (Goldberg and Kosorok, 2012; Huang and Ning, 2012; Huang et al., 2014), high-dimensional data (McKeague and Qian, 2013), missing data (Shortreed et al., 2014), multiple outcomes (Laber et al., 2014b; Linn et al., 2015), and restrictions on treatment resource (Luedtke and van der Laan, 2015).

Despite many estimation methods, none are designed to handle functionals of the response distribution other than the mean, such as quantiles. The median response is often of interest in studies where the outcome follows a skewed distribution, such as the total time a women spends in second stage labor (Zhang et al., 2012c). Using the potential outcomes framework (Rubin, 1974; Rosenbaum and Rubin, 1983), Zhang et al. (2012c) develop methods for estimating quantiles of the potential outcomes from observational data. However, they focus on comparing treatments at a single intervention time point rather than estimation of an optimal dynamic treatment regime. Structural nested distribution models (SNDMs) estimated using g-estimation facilitate estimation of point treatment effects on the cumulative distribution function of the outcome (Robins, 2000; Vansteelandt et al., 2014). Thus far, SNDMs have not been extended to estimate a regime that maximizes a quantile.

Q-learning and its variants are often useful when targeting an optimal regime because they provide relatively interpretable decision rules that are based on (typically linear) regression models. However, the Q-learning algorithm is an approximate dynamic programming procedure that requires modeling nonsmooth, nonmonotone transformations of data. This leads to nonregular estimators for parameters that index the optimal regime and complicates the search for models that fit the data well since many standard regression modeling diagnostics are invalid (Robins, 2004; Chakraborty et al., 2010; Laber et al., 2014c; Song et al., 2015). In addition, Q-learning with linear models does not target the globally optimal rule when the true conditional means are nonlinear. Interactive Q-learning (IQ-learning), developed for the two-stage binary treatment setting, requires modeling only smooth, monotone transformations of the data, thereby reducing problems of model misspecification and nonregular inference (Laber et al., 2014a). We extend the IQ-learning framework to optimize function-als of the outcome distribution other than the expected value. In particular, we optimize threshold-exceedance probabilities and quantiles of the response distribution. Furthermore, because this extension of IQ-learning provides an estimator of a threshold-exceedance probability or quantile of the response distribution under any postulated dynamic treatment regime, it can be used to construct direct-search estimators.

Threshold-exceedance probabilities are relevant in clinical applications where the primary objective is remission or a specific target for symptom reduction. For example, consider a population of obese patients enrolled in a study to determine the effects of several treatment options for weight loss. The treatments of interest may include combinations of drugs, exercise programs, counseling, and meal plans (Berkowitz et al., 2010). Our method can be used to maximize the probability that patients achieve a weight below some prespecified, patient-specific threshold at the conclusion of the study. Optimization of threshold-exceedance probabilities can be framed as a special case optimizing the mean of a binary outcome, for which several methods exist, including the classification-based Outcome Weighted Learning (Zhang et al., 2012b; Zhao et al., 2015). However, our approach is particularly useful for setting up the more challenging problem of quantile optimization.

With adjustments to our method of maximizing probabilities, we derive optimal decision rules for maximizing quantiles of the response distribution. Both frameworks can be used to study the entire distribution of the outcome under an optimal dynamic treatment regime; thus, investigators can examine how the optimal regime changes as the target probability or quantile is varied. In addition, the quantile framework provides an analog of quantile regression in the dynamic treatment regime setting for constructing robust estimators; for example, it enables optimization of the median response.

2. Generalized Interactive Q-Learning

We first characterize the optimal regime for a probability and quantile using potential outcomes (Rubin, 1974) and two treatment time-points. We assume that the observed data, $\mathcal{D}={\{({\mathit{X}}_{1i},A_{1i},{\mathit{X}}_{2i},A_{2i},Y_i)\}}_{i=1}^n$, comprise n independent, identically distributed, time-ordered trajectories; one per patient. Let (X₁, A₁, X₂, A₂, Y) denote a generic observation where: X₁ ∈ ℝ^p₁ is baseline covariate information collected prior to the first treatment; A₁ ∈ {−1, 1} is the first treatment; X₂ ∈ ℝ^p₂ is interim covariate information collected during the course of the first treatment but prior the second treatment; A₂ ∈ {−1, 1} is the second treatment; and Y ∈ ℝ is an outcome measured at the conclusion of stage two, coded so that larger is better. Define H₁ = X₁ and ${\mathit{H}}_2={({\mathit{H}}_1^{\mathbf{\top }},A_1,{\mathit{X}}_2^{\mathbf{\top }})}^{\mathbf{\top }}$ so that H_t is the information available to a decision maker at time t. A regime, π = (π₁, π₂), is a pair of decision rules where π_t : dom(H_t) ↦ dom(A_t), such that a patient presenting with H_t = h_t at time t is recommended treatment π_t(h_t).

Let ${\mathit{H}}_2^{\ast }(a_1)$ be the potential second-stage history under treatment a₁ and Y*(a₁, a₂) the potential outcome under treatment sequence (a₁, a₂). Define the set of all potential outcomes $W=\{{\mathit{H}}_2^{\ast }(a_1),Y\ast (a_1,a_2):(a_1,a_2)\in {\{-1,1\}}^2\}$. Throughout we assume: (C1) consistency, so that Y = Y*(A₁, A₂); (C2) sequential ignorability (Robins, 2004), i.e., A_t ╨ W | H_t for t = 1, 2; and (C3) positivity, so that there exists ε > 0 for which ε < pr(A_t = a_t|H_t) < 1 – ε with probability one for all a_t, t = 1, 2. Assumptions (C2)-(C3) hold by design when data are collected using a sequential multiple assignment randomized trial (SMART, Lavori and Dawson, 2000, 2004; Murphy, 2005b). In observational studies, these assumptions are not testable. We assume that data are collected using a two-stage, binary treatment SMART. This set-up facilitates a focused discussion of the proposed methods and is also useful in practice, as data in many sequentially randomized trials have this structure (Projects Using SMART, 2012; Laber, 2013). However, the following argument demonstrates that the proposed methodology can be extended to observational data and studies with more than two treatments.

For any π define $Y^{\ast }(\mathit{\pi })={\sum }_{(a_1,a_2)}Y\ast (a_1,a_2){\mathbb{1}}_{{\pi }_1({\mathit{H}}_1)=a_1}{\mathbb{1}}_{{\pi }_2\{{\mathit{H}}_2^{\ast }(a_1)\}=a_2}$ to be the potential outcome under π. Define the function R(y; x₁, a₁, x₂, a₂) = pr(Y > y|X₁ = x₁, A₁ = a₁, X₂ = x₂, A₂ = a₂) for any y ∈ ℝ. Assuming (C1)-(C3) and y ∈ ℝ, the survival function of Y*(π) can be expressed in terms of the underlying generative model as

$$\text{pr}\{Y\ast (\mathit{\pi })>y\}=\mathbb{E}\left[\sum _{a_1}{\mathbb{1}}_{{\mathit{\pi }}_1({\mathit{H}}_1)=a_1}\mathbb{E}\right\{\sum _{a_2}{\mathbb{1}}_{{\mathit{\pi }}_2({\mathit{H}}_2)=a_2}R(y;{\mathit{X}}_1,a_1,{\mathit{X}}_2,a_2)|{\mathit{X}}_1,A_1=a_1\}]$$

for any π (Robins, 1986). This result shows that pr{Y*(π) > y} is maximized by the regime ${\mathit{\pi }}^y=({\pi }_1^y,{\pi }_2^y)$, where ${\pi }_2^y({\mathit{h}}_2)arg{max}_{a_2}R(y;{\mathit{x}}_1,a_1,{\mathit{x}}_2,a_2)$ and ${\pi }_1^y({\mathit{h}}_1)=arg{max}_{a_1}\mathbb{E}\{{\sum }_{a_2}{\mathbb{1}}_{{\pi }_2^y({\mathit{H}}_2)=a_2}R(y;{\mathit{x}}_1,a_1,{\mathit{X}}_2,a_2)|{\mathit{X}}_1={\mathit{x}}_1,A_1=a_1\}$. This result can also be used to characterize the regime that optimizes a quantile. For any regime π, pr{Y*(π^y) ≤ y} ≤ pr{Y*(π) ≤ y} implies inf{y : pr{Y*(π^y) ≤ y} ≥ τ} ≥ inf{y : pr{Y*(π) ≤ y} ≥ τ}. In fact the left-hand side is the optimal τ^th quantile; denote it by $y_{\tau }^{\ast }$. An optimal regime with respect to the τ^th quantile is thus any regime that, when used to assign treatments in the population, results in a τ^th quantile that attains $y_{\tau }^{\ast }$. The following result is proved in the supplemental material.

Theorem 2.1. Let ε > 0 and τ ∈ (0, 1) be arbitrary but fixed. Assume (C1)-(C3) and that the map y ↦ R(y; x₁, a₁, x₂, a₂) from ℝ into (0, 1) is continuous and strictly increasing in a neighborhood of τ for all x₁, a₁, x₂, and a₂. Then, $inf\{y:\mathit{\text{pr}}\{Y^{\ast }({\mathit{\pi }}^{y_{\tau }^{\ast }})\le y\}\ge \tau \}=y_{\tau }^{\ast }$.

Theorem 2.1 states that the regime ${\mathit{\pi }}^{y_{\tau }^{\ast }}$ induces a τ^th quantile of the potential outcome distribution that attains the optimal τ^th quantile. Without the strictly increasing assumption, ${\mathit{\pi }}^{y_{\tau }^{\ast }}$ may not be optimal. However, it can be shown that there exists a value ỹ ∈ ℝ such that the regime π^ỹ attains a τ^th quantile that is arbitrarily close to $y_{\tau }^{\ast }$. Details are given in Section 3.

2.1 Threshold Interactive Q-learning

The estimators presented in this section serve as useful building blocks for developing the estimators in the next section which focus on quantile optimization. Here, we derive and estimate the optimal set of decision rules for maximizing a threshold-exceedance probability. Let pr^π(Y > λ), equivalently pr^{π₁, π₂}(Y > λ), denote the probability that the outcome Y is greater than a predefined threshold λ under treatment assignment dictated by the regime π = (π₁, π₂). Threshold Interactive Q-learning (TIQ-learning) maximizes pr^π{Y > λ(H_t) | H₁} for all H₁ with respect to π, where λ(H_t) is a threshold that depends on H_t, t = 1, 2. Here, we assume λ(H_t) ≡ λ; patient-specific thresholds are discussed in the supplemental material.

As pr^π(Y > λ) = E^π(𝟙_Y>λ), many approaches exist for estimating an optimal regime that maximizes the value function of the binary outcome 𝟙_Y>λ; one example is discrete Q-learning (Chakraborty and Moodie, 2013). We show analytically in Remark 2.4, and empirically in Section 4, that discrete Q-learning using the logit link is equivalent to Q-learning with outcome Y and is therefore insensitive to the threshold λ. Nonparametric methods such as OWL (Zhao et al., 2012) double-robust direct search (Zhang et al., 2012b,a, 2013), and boosting (Kang et al., 2014) are less restrictive than model-based approaches for optimizing binary outcomes, but the following exposition facilitates later developments for optimizing quantiles.

Our estimators are derived under the following set-up. Because A₂ is binary, there exist functions m and c such that E(Y | A₂, H₂) = m(H₂) + A₂c(H₂). We assume that Y = E(Y | A₂, H₂) + ε, where E(ε) = 0, Var(ε) = σ², and ε is independent of (A₂, H₂). In the supplemental material we describe extensions to: (i) heteroskedastic error structures, where Y = E(Y | A₂, H₂) + σ(H₂, A₂)ε for unknown function σ; and (ii) non-additive error structures such as the multiplicative error model, Y = ε[m(H₂) + A₂c(H₂)], provided ε > 0 with probability one and pr{m(H₂) + A₂c(H₂) = 0} = 0.

Define F_H1(·) to be the distribution of H₁; F_{H2|H1, A1}(· | h₁, a₁) to be the conditional distribution of H₂ given H₁ = h₁ and A₁ = a₁; F_ε(·) to be the distribution of ε; and ${\mathit{H}}_2^{{\pi }_1({\mathit{H}}_1)}={\{{\mathit{H}}_1^{\mathbf{\top }},{\pi }_1({\mathit{H}}_1),{\mathit{X}}_2^{\mathbf{\top }}\}}^{\mathbf{\top }}$. Let $J^{{\pi }_1,{\pi }_2}({\mathit{h}}_1,{\mathit{h}}_2,y)=F_{\epsilon }\{y-m({\mathit{h}}_2^{{\pi }_1({\mathit{h}}_1)})-{\pi }_2({\mathit{h}}_2^{{\pi }_1({\mathit{h}}_1)})c({\mathit{h}}_2^{{\pi }_1({\mathit{h}}_1)})\}$, then

$${\text{pr}}^{{\pi }_1,{\pi }_2}(Y\le y)=\int \int J^{{\pi }_1,{\pi }_2}({\mathit{h}}_1,{\mathit{h}}_2,y)d{F}_{{\mathit{H}}_2|{\mathit{H}}_1,A_1}\{{\mathit{h}}_2|{\mathit{h}}_1,{\pi }_1({\mathit{h}}_1)\}d{F}_{{\mathit{H}}_1}({\mathit{h}}_1),$$ (1)

is the expected value of J^{π₁, π₂}(H₁, H₂, y).

Let ${\pi }_2^{\ast }({\mathit{h}}_2)=sgn\{c({\mathit{h}}_2)\}$, where sgn(x) = 𝟙_x≥0 – 𝟙_x<0. Then, $J^{{\pi }_1,{\pi }_2^{\ast }}({\mathit{h}}_1,{\mathit{h}}_2,y)=F_{\epsilon }\{y-m({\mathit{h}}_2^{{\pi }_1({\mathit{h}}_1)})-|c({\mathit{h}}_2^{{\pi }_1({\mathit{h}}_1)})|\}$ and ${\pi }_2({\mathit{h}}_2^{{\pi }_1({\mathit{h}}_1)})c({\mathit{h}}_2^{{\pi }_1({\mathit{h}}_1)})\le |c({\mathit{h}}_2^{{\pi }_1({\mathit{h}}_1)})|$ for all ${\mathit{h}}_2^{{\pi }_1({\mathit{h}}_1)}$ implies

$${\text{pr}}^{{\pi }_1,{\pi }_2}(Y\le y)\ge \int \int J^{{\pi }_1,{\pi }_2^{\ast }}({\mathit{h}}_1,{\mathit{h}}_2,y)d{F}_{{\mathit{H}}_2|{\mathit{H}}_1,A_1}\{{\mathit{h}}_2|{\mathit{h}}_1,{\pi }_1({\mathit{h}}_1)\}d{F}_{{\mathit{H}}_1}({\mathit{h}}_1),$$ (2)

where the right-hand side of (2) is ${\text{pr}}^{{\pi }_1,{\pi }_2^{\ast }}(Y\le y)$. Let G(·, · | h₁, a₁) denote the joint conditional distribution of m(H₂) and c(H₂) given H₁ = h₁ and A₁ = a₁, then ${\text{pr}}^{{\pi }_1,{\pi }_2^{\ast }}(Y\le y)=E(I[y,F_{\epsilon }(\cdot ),G\{\cdot ,\cdot |{\mathit{H}}_1,{\pi }_1({\mathit{H}}_1)\}])$, where

$$I\{y,F_{\epsilon }(\cdot ),G(\cdot ,\cdot |{\mathit{h}}_1,a_1)\}=\int F_{\epsilon }(y-u-|v|)dG(u,v|{\mathit{h}}_1,a_1).$$ (3)

The λ-optimal regime ${\mathit{\pi }}_{\lambda }^{\text{TIQ}}=\{{\pi }_{1,\lambda }^{\text{TIQ}},{\pi }_{2,\lambda }^{\text{TIQ}}\}$ satisfies ${\text{pr}}^{{\mathit{\pi }}_{\lambda }^{\text{TIQ}}}(Y>\lambda )\ge {\text{pr}}^{\mathit{\pi }}(Y>\lambda )$ for all π. That is, the distribution of Y induced by regime ${\mathit{\pi }}_{\lambda }^{\text{TIQ}}$ has at least as much mass above λ as the distribution of Y induced by any other regime. It follows from the lower bound on pr^{π₁, π₂}(Y ≤ y) displayed in (2) that ${\pi }_{2,\lambda }^{\text{TIQ}}({\mathit{h}}_2)={\pi }_2^{\ast }({\mathit{h}}_2)=sgn\{c({\mathit{h}}_2)\}$ for all h₂, independent of λ and ${\pi }_{1,\lambda }^{\text{TIQ}}$. Henceforth, we denote ${\pi }_{2,\lambda }^{\text{TIQ}}$ by ${\pi }_2^{\ast }$. The relationship

$${\text{pr}}^{{\pi }_1,{\pi }_2^{\ast }}(Y>\lambda )=1-E(I[\lambda ,F_{\epsilon }(\cdot ),G\{\cdot ,\cdot |{\mathit{H}}_1,{\pi }_1({\mathit{H}}_1)\}])\le 1-E[\underset{a_1}{min}I\{\lambda ,F_{\epsilon }(\cdot ),G(\cdot ,\cdot |{\mathit{H}}_1,a_1)\}],$$ (4)

shows that the λ-optimal first-stage rule is ${\pi }_{1,\lambda }^{\text{TIQ}}({\mathit{h}}_1)=\underset{a_1}{arg\; min}I\{\lambda ,F_{\epsilon }(\cdot ),G(\cdot ,\cdot |{\mathit{h}}_1,a_1)\}$. Inequality (4) holds because I{λ, F_ε(·), G(·, · | H₁, a₁)} is minimized over a₁ for all H₁. It will be useful later on to write ${\pi }_{1,\lambda }^{\text{TIQ}}({\mathit{h}}_1)=sgn\{d({\mathit{h}}_1,\lambda )\}$ where

$$d({\mathit{h}}_1,\lambda )=I\{\lambda ,F_{\epsilon }(\cdot ),G(\cdot ,\cdot |{\mathit{h}}_1,-1)\}-I\{\lambda ,F_{\epsilon }(\cdot ),G(\cdot ,\cdot |{\mathit{h}}_1,1)\}.$$ (5)

Below, we describe the general form of the TIQ-learning algorithm that can be used to estimate the λ-optimal regime. The exact algorithm depends on the choice of estimators for m(H₂), c(H₂), F_ε(·) and G(·, · | h₁, a₁). For example, one might posit parametric models, m(H₂; β_2,0) and c(H₂; β_2,1), for m(H₂) and c(H₂) and estimate the parameters in the model Y = m(H₂; β_2,0) + c(H₂; β_2,1) + ε using least squares. Alternatively, these terms could be estimated nonparametrically. We discuss possible estimators for F_ε(·) and G(·, ·| h₁, a₁) in Sections 3.1 and 3.2. In practice, the choice of estimators should be informed by the observed data. Finally, we emphasize that if the binary threshold outcome is the terminal focus, rather than quantile optimization, a different approach (e.g., nonparametric or direct search) may be warranted. The following algorithm provides a foundation for the quantile optimization algorithm in the next section. Define d̂(h₁, λ) = I{λ, F̂_ε(·), Ĝ(·, · | h₁, −1)} – I{λ, F̂_ε(·), Ĝ(·, · | h₁, 1)}.

TIQ-learning algorithm

• TIQ.1 Estimate m(H₂) and c(H₂), and denote the resulting estimates by m̂(H₂) and ĉ(H₂). Given h₂, estimate ${\pi }_2^{\ast }$ using the plug-in estimator ${\widehat{\pi }}_2^{\ast }({\mathit{h}}_2)=sgn\{\widehat{c}({\mathit{h}}_2)\}$.

• TIQ.2 Estimate F_ε(·), the cumulative distribution function of ε, using the residuals ê^Y = Y – m̂(H₂) – A₂ĉ(H₂) from TIQ.1. Let F̂_ε(·) denote this estimator.

• TIQ.3 Estimate G(·, · | h₁; a₁), the joint conditional distribution of m(H₂) and c(H₂) given H₁ = h₁ and A₁ = a₁. Let Ĝ(·, · | h₁, a₁) denote this estimator.

• TIQ.4 Given h₁, estimate ${\pi }_{1,\lambda }^{\text{TIQ}}$ using the plug-in estimator ${\widehat{\pi }}_{1,\lambda }^{\text{TIQ}}({\mathit{h}}_1)=sgn\{\widehat{d}({\mathit{h}}_1,\lambda )\}$.

The TIQ-learning algorithm involves modeling m(H₂), c(H₂), the distribution function F_ε(·), and the bivariate conditional density G(·, · | h₁, a₁). This is more modeling than some mean-targeting algorithms. For example, Q-learning requires modeling m(H₂), c(H₂), and the conditional mean of m(H₂) + |c(H₂)| given H₁, A₁, while IQ-learning requires modeling m(H₂), c(H₂), the conditional mean of m(H₂), and the conditional density of c(H₂) (Laber et al., 2014a). We discuss models for the components of the TIQ-learning algorithm in Sections 3.1 and 3.2.

Remark 2.2. Standard trade-offs between parametric and nonparametric estimation apply to all terms in the TIQ-learning algorithm. In practice, the choice of estimators will likely depend on sample size and the scientific goals of the study. If the goal is to estimate a regime for immediate decision support in the clinic, then the marginal mean outcome of the estimated regime is of highest priority. Given sufficient data, it may be desirable to use nonparametric estimators in this context. However, if the goal is to inform future research and generate hypotheses for further investigation, then factors like parsimony, interpretability, and the ability to identify and test for factors associated with heterogeneous treatment response may be most important. In this context, parametric models may be preferred for components of the TIQ-learning algorithm, namely, the treatment interaction term c(H₂), while more flexible models may be specified for the nuisance function m(H₂). A parsimonious, parametric model for c(H₂) trades a restriction on the class of possible regimes for potential gains in interpretability. Note that approaches such as OWL (Zhao et al., 2012) and the methods described in Luedtke and van der Laan (2015) do not require models for m(H₂), while double robust approaches model this term only to increase efficiency and remain consistent for the optimal regime even when the model is misspecified.

Remark 2.3. Let ${\pi }_1^{\mathrm{M}}$ denote the first-stage decision rule of an optimal regime for the mean of Y. Then, assuming the set-up of Section 2.1, it can be shown that

$${\pi }_1^{\mathrm{M}}({\mathit{h}}_1)=arg\underset{a_1}{min}\int (-u-|v|)dG(u,v|{\mathit{h}}_1,a_1)=arg\underset{a_1}{min}\int (\lambda -u-|v|)dG(u,v|{\mathit{h}}_1,a_1),$$

whereas ${\pi }_{1,\lambda }^{\text{TIQ}}({\mathit{h}}_1)=arg{min}_{a_1}\int F_{\epsilon }(\lambda -u-|v|)dG(u,v|{\mathit{h}}_1,a_1)$. If F_ε(·) is approximately linear where the conditional distribution of λ – m(H₂) – |c(H₂)| given H₁ = h₁ and A₁ = a₁ is concentrated, ${\pi }_1^{\mathrm{M}}({\mathit{h}}_1)$ and ${\pi }_{1,\lambda }^{\text{TIQ}}({\mathit{h}}_1)$ will likely agree. Thus, the difference between the mean optimal and TIQ-learning optimal regimes can be compared empirically by computing arg min_a1 ∫(−u – |v|)dĜ(u, v | h_1i, a₁), arg min_a1 ∫ F̂_ε(λ – u – |v|)dĜ(u, v | h_1i, a₁), for each first-stage patient history h_1i, i = 1, …, n, and examining where these rules differ.

Remark 2.4. One approach to estimating an optimal decision rule for threshold-exceedance probabilities is discrete Q-learning (Chakraborty and Moodie, 2013). Suppose Y = m*(H₂)+A₂c*(H₂) + ε, where ε has cumulative distribution function F_ε(·). For the binary outcome 𝟙_Y>λ, define the second-stage Q-function, Q₂(H₂, A₂) = pr(𝟙_Y>λ = 1 | H₂, A₂), and the first stage Q-function, Q₁(H₁, A₁) = 𝔼[max_a2 Q₂(H₂, a₂) | H₁, A₁]. If these functions were known, the optimal treatment assignments for the set of observed histories (H₁ = h₁, H₂ = h₂) would be {arg max_a1 Q₁(h₁, a₁), arg max_a2 Q₂(h₂, a₂)}. In practice, the Q-functions are unknown and the Q-learning algorithm proceeds by specifying models for them. Denote estimates of the Q-functions obtained from such models by Q̂₂(H₂, A₂) and Q̂₁(H₁, A₁). Then, the estimated optimal treatment assignments for (H₁ = h₁, H₂ = h₂) are {arg max_a1 Q̂₁(h₁, a₁), arg max_a2 Q̂₂(h₂, a₂)}. For binary outcomes, logistic regression is often a natural model choice for Q₂. Subsequently, at the first stage one would specify a model for 𝔼[max_a2 Q̂₂(H₂, a₂) | H₁, A₁], where the pseudo outcome max_a2 Q̂₂(H₂, a₂) is bounded in [0, 1]. Rather than modeling this conditional expectation with linear regression, which may result in Q̂₁ estimates outside the interval [0, 1], an alternative is to model 𝔼[max_a2 logit{Q̂₂(H₂, a₂)} | H₁, A₁] using linear regression, since max_a2 logit{Q̂₂(H₂, a₂)} takes values on the real line (Chakraborty and Moodie, 2013; Moodie et al., 2014). However, for our special case of discrete Q-learning with outcome 𝟙_Y>λ, the logit link function is misspecified for pr(𝟙_Y>λ = 1 | H₂, A₂). The correct link function for the second stage Q-function is $L(u)=\lambda -F_{\epsilon }^{-1}(1-u)$. When the logit link is used rather than L(u), discrete Q-learning as described above and Q-learning for the continuous outcome Y perform similarly across all values of λ for the generative model for Y given above. We demonstrate this using simulation experiments in Section 4.

3. Quantile Interactive Q-Learning

Under some generative models, assigning treatment according to a mean-optimal regime leads to higher average outcomes at the expense of higher variability, negatively affecting patients with outcomes in the lower quantiles of the induced distribution of Y. We demonstrate this using simulated examples in Section 4. Define the τ^th quantile of the distribution of Y induced by regime π as q^π(τ) = inf{y : pr^π₁,π₂(Y ≤ y) ≥ τ}. The goal of Quantile Interactive Q-learning (QIQ-learning) is to estimate a pair of decision rules, ${\mathit{\pi }}_{\tau }^{\text{QIQ}}=\{{\pi }_{1,\tau }^{\text{QIQ}},{\pi }_{2,\tau }^{\text{QIQ}}\}$, that maximize q^π(τ) over π for a fixed, prespecified τ. QIQ-learning is similar to TIQ-learning, but the optimal first-stage rule is complicated by the inversion of the distribution function to obtain quantiles of Y under a given regime. When the variance of Y is independent of A₂, the QIQ-learning second-stage optimal decision is ${\pi }_{2,\tau }^{\text{QIQ}}({\mathit{h}}_2)={\pi }_2^{\ast }({\mathit{h}}_2)=sgn\{c({\mathit{h}}_2)\}$, independent of τ and ${\pi }_{1,\tau }^{\text{QIQ}}$; details are provided in Section 4 of the supplemental material. Denote ${\pi }_{2,\tau }^{\text{QIQ}}$ by ${\pi }_2^{\ast }$.

Next we characterize ${\pi }_{1,\tau }^{\text{QIQ}}$, which will motivate an algorithm for calculating it. Let d(h₁, y) be as in (5), and define Γ(h₁, y) ≜ sgn{d(h₁, y)}. Then $\mathrm{\Gamma }({\mathit{h}}_1,y)={\pi }_{1,\lambda }^{\text{TIQ}}({\mathit{h}}_1){|}_{\lambda =y}$ is the optimal first-stage decision rule of TIQ-learning at λ = y. We have introduced the new notation to emphasize the dependence on y. Next, define the optimal τ^th quantile

$$y_{\tau }^{\ast }\triangleq inf\{y:{\text{pr}}^{\mathrm{\Gamma }(·,y),{\pi }_2^{\ast }}(Y\le y)\ge \tau \},$$ (6)

which we study further in the remainder of this section.

Lemma 7.7 of the supplemental material proves that ${lim}_{y\to \infty (-\infty )}{\text{pr}}^{\mathrm{\Gamma }(·,y),{\mathit{\pi }}_2^{\ast }}(Y\le y)=1(0)$, so that $y_{\tau }^{\ast }$ is defined for all τ ∈ (0, 1). For each y ∈ ℝ,

$${\text{pr}}^{{\pi }_1,{\pi }_2^{\ast }}(Y\le y)=E(I[y,F_{\epsilon }(·),G\{·,·|{\mathit{H}}_1,{\pi }_1({\mathit{H}}_1)\}])\ge E(I[y,F_{\epsilon }(·),G\{·,·|{\mathit{H}}_1,\mathrm{\Gamma }({\mathit{H}}_1,y)\}])={\text{pr}}^{\mathrm{\Gamma }(·,y),{\pi }_2^{\ast }}(Y\le y),$$

where I(·, ·, ·) is defined in (3). The last equality follows because Γ(H₁, y) minimizes E (I [y, F_ε(·), G{·, ·|H₁, a₁}]) with respect to a₁. Hence, $\{y:{\text{pr}}^{\mathrm{\Gamma }(·,y),{\pi }_2^{\ast }}(Y\le y)\ge \tau \}\subseteq \{y:{\text{pr}}^{{\pi }_1,{\pi }_2^{\ast }}(Y\le y)\ge \tau \}$, and taking the infimum on both sides gives the upper bound

$$y_{\tau }^{\ast }\ge q^{{\pi }_1,{\pi }_2^{\ast }}(\tau )\text{for all}{\pi }_1.$$ (7)

Thus, a first-stage decision rule π₁ is optimal if it induces a τ^th quantile equal to the upper bound $y_{\tau }^{\ast }$ when treatments are subsequently assigned according to ${\pi }_{\tau }^{\ast }$, i.e., if $q^{{\pi }_1,{\pi }_2^{\ast }}(\tau )=y_{\tau }^{\ast }$.

We now discuss conditions that guarantee existence of a π₁ such that $q^{{\pi }_1,{\pi }_2^{\ast }}(\tau )=y_{\tau }^{\ast }$ and derive its form. The quantile obtained under regime $\pi =\{\mathrm{\Gamma }(·,y),{\pi }_2^{\ast }\}$ is

$$f(y)\triangleq q^{\mathrm{\Gamma }(·,y),{\pi }_2^{\ast }}(\tau )=inf\{\stackrel{\sim }{y}:{\text{pr}}^{\mathrm{\Gamma }(·,y),{\pi }_2^{\ast }}(Y\le \stackrel{\sim }{y})\ge \tau \}.$$ (8)

Thus, because it is a quantile and the bound in (7) applies, ${\text{pr}}^{\mathrm{\Gamma }(·,y),{\pi }_2^{\ast }}\{Y\le f(y)\}\ge \tau$, and $f(y)=q^{\mathrm{\Gamma }(·,y),{\pi }_2^{\ast }}(\tau )\le y_{\tau }^{\ast }$ for all y. Our main results depend on the following lemma, which is proved in the supplemental material.

Lemma 3.1.

$$(A)y<y_{\tau }^{\ast }\mathit{\text{implies}}y<f(y)\le y_{\tau }^{\ast };$$ (9)

$$(B)f(y_{\tau }^{\ast -})\triangleq \underset{\delta \downarrow 0}{lim}f(y_{\tau }^{\ast }-\delta )=y_{\tau }^{\ast };$$ (10)

$$(C)f(y_{\tau }^{\ast })\le y_{\tau }^{\ast }\mathit{\text{with strict inequality if there exists}}\delta >0\mathit{\text{such that}}{\mathit{\text{pr}}}^{\mathrm{\Gamma }(·,y),{\pi }_2^{\ast }}(Y\le y_{\tau }^{\ast }-\delta )\ge \tau ;$$ (11)

$$(D)\mathit{\text{If}}{F}_{\epsilon }(·)\mathit{\text{is continuous and strictly increasing}},\mathit{\text{then}}f(y_{\tau }^{\ast })=y_{\tau }^{\ast }.$$ (12)

It follows from (B) that $f(y_{\tau }^{\ast })=y_{\tau }^{\ast }$ if and only if f(y) is left continuous at $y=y_{\tau }^{\ast }$, and part (D) is a sufficient condition guaranteeing left-continuity of f(y) at $y_{\tau }^{\ast }$. In this case, the optimal first-stage rule is ${\pi }_{1,\tau }^{\text{QIQ}}({\mathit{h}}_1)=\mathrm{\Gamma }({\mathit{h}}_1,y_{\tau }^{\ast })$, i.e., $q^{\mathrm{\Gamma }(·,y_{\tau }^{\ast }),{\pi }_2^{\ast }}(\tau )=y_{\tau }^{\ast }$. The condition stated in (D) is commonly satisfied, e.g., when the density of ε has positive support on the entire real line. If f(y) is not left continuous at $y_{\tau }^{\ast }$, and thus $f(y_{\tau }^{\ast })=y_{\tau }^{\ast }$, in light of (10) we can always approach the optimal policy via a sequence of regimes of the form { $\mathrm{\Gamma }(·,y_{\tau }^{\ast }-{\delta }_n),{\pi }_2^{\ast }$}, where δ_n decreases to 0. If the underlying distributions of the histories and Y were known, the following algorithm produces an optimal regime.

Population-level algorithm to find ${\pi }_{1,\tau }^{\text{QIQ}}$

1. Compute $y_{\tau }^{\ast }$ from (6) and $f(y_{\tau }^{\ast })$ from (8).

2. 1. If $f(y_{\tau }^{\ast })=y_{\tau }^{\ast }$, ${\pi }_{1,\tau }^{\text{QIQ}}({\mathit{h}}_1)=\mathrm{\Gamma }({\mathit{h}}_1,y_{\tau }^{\ast })$ is optimal as it attains the quantile $y_{\tau }^{\ast }$.
   2. If $f(y_{\tau }^{\ast })<y_{\tau }^{\ast }$, ${\pi }_{1,\tau }^{\text{QIQ}}({\mathit{h}}_1)={lim}_{\delta \downarrow 0}\mathrm{\Gamma }({\mathit{h}}_1,y_{\tau }^{\ast }-\delta )$ is optimal. Note that this rule can be written in closed form as ${lim}_{\delta \downarrow 0}\mathrm{\Gamma }({\mathit{h}}_1,y_{\tau }^{\ast }-\delta )=sgn\{d^{-}({\mathit{h}}_1,y_{\tau }^{\ast })\}$, where we define $d^{-}({\mathit{h}}_1,y_{\tau }^{\ast })=I_{\epsilon }(a_1){|}_{a_1=-1}-I_{\epsilon }(a_1){|}_{a_1=1}$ and $I_{\epsilon }(a_1)=\int \text{pr}(\epsilon <y_{\tau }^{\ast }-u-|v|)dG(u,v|{\mathit{h}}_1,a_1).$.

In practice, the generative model is not known, but the population-level algorithm suggests an estimator of ${\pi }_{1,\tau }^{\text{QIQ}}$. The following QIQ-learning algorithm can be used to estimate an optimal first-stage decision rule. The exact algorithm depends on the choice of estimators for F_ε(·) and G(·, · | h₁, a₁); several options are presented in Sections 3.1 and 3.2, but the choice should be data-driven; see, e.g., Remark 2.2.

QIQ-learning algorithm

• QIQ.1 Follow TIQ.1 – TIQ.3 of the TIQ-learning algorithm in Section 2.1.

• QIQ.2 With I(·, ·, ·) as in (3) and first-stage patient histories h_1i, estimate $y_{\tau }^{\ast }$ using

  $${\widehat{y}}_{\tau }^{\ast }\triangleq inf(y:\frac{1}{n}\sum _{i=1}^{n}I[y,{\widehat{F}}_{\epsilon }(·),\widehat{G}\{·,·|{\mathit{h}}_{1i},\widehat{\mathrm{\Gamma }}({\mathit{h}}_{1i},y)\}]\ge \tau ).$$
• QIQ.3 Estimate $f(y_{\tau }^{\ast })$ using

  $$\widehat{f}({\widehat{y}}_{\tau }^{\ast })\triangleq inf(y:\frac{1}{n}\sum _{i=1}^{n}I[y,{\widehat{F}}_{\epsilon }(·),\widehat{G}\{·,·|{\mathit{h}}_{1i},\widehat{\mathrm{\Gamma }}({\mathit{h}}_{1i},{\widehat{y}}_{\tau }^{\ast })\}]\ge \tau ).$$
• QIQ.4

  1. If $\widehat{f}({\widehat{y}}_{\tau }^{\ast })={\widehat{y}}_{\tau }^{\ast }$, then ${\widehat{\pi }}_{1,\tau }^{\text{QIQ}}({\mathit{h}}_1)=\widehat{\mathrm{\Gamma }}({\mathit{h}}_1,{\widehat{y}}_{\tau }^{\ast })$ is an estimated optimal first-stage decision rule because it attains the estimated optimal quantile, ${\widehat{y}}_{\tau }^{\ast }$, when treatments are subsequently assigned according to ${\pi }_2^{\ast }$ at the second stage.
  2. If $\widehat{f}({\widehat{y}}_{\tau }^{\ast })={\widehat{y}}_{\tau }^{\ast }$, then the first-stage rule ${\widehat{\pi }}_{1,\tau }^{\text{QIQ}}({\mathit{h}}_1)=\widehat{\mathrm{\Gamma }}({\mathit{h}}_1,{\widehat{y}}_{\tau }^{\ast }-\delta )$, δ > 0, results in the estimated quantile $\widehat{f}({\widehat{y}}_{\tau }^{\ast }-\delta )$, which satisfies ${\widehat{y}}_{\tau }^{\ast }-\delta <\widehat{f}({\widehat{y}}_{\tau }^{\ast }-\delta )\le {\widehat{y}}_{\tau }^{\ast }$. By choosing δ arbitrarily small, this estimated quantile will be arbitrarily close to the estimated optimal quantile ${\widehat{y}}_{\tau }^{\ast }.$.

To complete the TIQ- and QIQ-learning algorithms, we provide specific estimators F_ε(·) and G(·, · | h₁, a₁) in the next two sections. We suggest estimators that are likely to be useful in practice, but our list is not exhaustive. An advantage of TIQ- and QIQ-learning is that they involve modeling only smooth transformations of the data; these are standard, well-studied modeling problems in the statistics literature.

3.1 Working models for F_ε(·)

Both TIQ- and QIQ-learning require estimation of the distribution function of the second-stage error, ε. We suggest two estimators that are useful in practice. The choice between them can be guided by inspection of the residuals from the second-stage regression.

Normal Scale Model

The normal scale estimator for F_ε(·) is ${\widehat{F}}_{\epsilon }^N(z)\triangleq \mathrm{\Phi }(z/{\widehat{\sigma }}_{\epsilon })$, where Φ(·) denotes the standard normal distribution function and σ̂_ε is the standard deviation of the second-stage residuals, ${\widehat{e}}_i^Y\triangleq Y_i-m({\mathit{H}}_{2i})-A_{2i}c({\mathit{H}}_{2i})$, i = 1, …, n. If it is thought that σ_ε depends on (H₂, A₂), flexibility can be gained by assuming a heteroskedastic variance model (Carroll and Ruppert, 1988), i.e., by assuming F_ε(z) = Φ{z/σ_ε(H₂, A₂)} for some unknown function σ_ε(h₂, a₂). Given an estimator σ̂_ε(h₂, a₂) of σ_ε(h₂, a₂), an estimator of F_ε(·) is ${\widehat{F}}_{\epsilon }^N(z)\triangleq \mathrm{\Phi }\{z/{\widehat{\sigma }}_{\epsilon }({\mathit{H}}_2,A_2)\}$. We discuss variance modeling techniques in the next section.

Nonparametric Model

For more flexibility, a non- or semi-parametric estimator for F_ε(·) can be used. In the homogeneous variance case, a nonparametric estimator of F_ε(·) is the empirical distribution of the residuals, ${\widehat{F}}_{\epsilon }^E(z)\triangleq n^{-1}{\sum }_{i=1}^n\mathbb{1}({\widehat{e}}_i^Y\le z)$. In the heterogeneous variance case, one can assume a non- or semi-parametric scale model F_ε|H2,A2(z|H₂ = h₂, A₂ = a₂) = F₀{z/σ_ε(h₂, a₂)}, where F₀(·) is an unspecified distribution function. Given an estimator σ̂_ε(h₂, a₂) of σ_ε(h₂, a₂), an estimator of F_ε|H2,A2(z|H₂ = h₂, A₂ = a₂) is ${\widehat{F}}_{\epsilon }^E(z|{\mathit{H}}_2={\mathit{h}}_2,A_2=a_2)=n^{-1}{\sum }_{i=1}^n\mathbb{1}\{{\stackrel{\sim }{e}}_i^Y\le z/{\widehat{\sigma }}_{\epsilon }({\mathit{h}}_2,a_2)\}$, where ${\stackrel{\sim }{e}}_i^Y={\stackrel{\sim }{e}}_i^Y/{\widehat{\sigma }}_{\epsilon }({\mathit{H}}_{2i},A_{2i})$. Standard residual diagnostic techniques, e.g., a normal quantile-quantile plot, can be used to determine whether a normal assumption seems plausible for the observed data.

3.2 Working models for G(·, ·|h₁, a₁)

In addition to a model for F_ε(·), TIQ- and QIQ-learning require models for the bivariate conditional density of m(H₂) and c(H₂) given H₁ and A₁. A useful strategy is to first model the conditional mean and variance functions of m(H₂) and c(H₂) and then estimate the joint distribution of their standardized residuals. Define these standardized residuals as

$$\begin{array}{cc}{e}^m=\frac{m({\mathit{H}}_2)-{\mu }_m({\mathit{H}}_1,A_1)}{{\sigma }_m({\mathit{H}}_1,A_1)},& e^c=\frac{c({\mathit{H}}_2)-{\mu }_c({\mathit{H}}_1,A_1)}{{\sigma }_c({\mathit{H}}_1,A_1)},\end{array}$$

where μ_m(H₁, A₁) ≜ E{m(H₂) | H₁, A₁} and ${\sigma }_m^2({\mathit{H}}_1,A_1)\triangleq E[{\{m({\mathit{H}}_2)-{\mu }_m({\mathit{H}}_1,A_1)\}}^2|{\mathit{H}}_1,A_1]$. The mean and variance functions of c(H₂) are defined similarly: μ_c(H₁, A₁) ≜ E{c(H₂)|H₁, A₁}, and ${\sigma }_c^2({\mathit{H}}_1,A_1)\triangleq E[{\{c({\mathit{H}}_2)-{\mu }_c({\mathit{H}}_1,A_1)\}}^2|{\mathit{H}}_1,A_1]$. In simulations, we use parametric mean and variance models for μ_m, ${\sigma }_m^2$, μ_c, and ${\sigma }_c^2$, and we estimate the joint distribution of e^m and e^c using a Gaussian copula. Alternatively, the joint residual distribution could be modelled parametrically, e.g., with a multivariate normal model; or nonparametrically, e.g., using a bivariate kernel density estimator (Silverman, 1986, Ch. 4). The Gaussian copula is used in the simulations in Section 4, and results are provided using a bivariate kernel estimator in the supplemental material. Common exploratory analysis techniques can be used to interactively guide the choice of estimator for G(·, · | h₁, a₁). In simulated experiments described in the supplemental material, a bivariate kernel density estimator was competitive with a correctly specified Gaussian copula model with sample sizes as small as n = 100. Using parametric mean and variance modeling, the following steps would be substituted in Step TIQ.3 of the TIQ-learning algorithm.

Mean and Variance Modeling

• 3.1 Compute ${\widehat{\mathit{\theta }}}_m\triangleq arg{min}_{{\mathit{\theta }}_m}{\sum }_{i=1}^n{\{\widehat{m}({\mathit{H}}_{2i})-{\mu }_m(H_{1i},A_{1i};{\mathit{\theta }}_m)\}}^2$ and the resulting estimator μ_m(H₁, A₁; θ̂_m) of the mean function μ_m(H₁, A₁).

• 3.2 Use the estimated mean function from Step 3.1 to obtain

  $${\widehat{\mathit{\gamma }}}_m\triangleq \underset{{\mathit{\gamma }}_m}{argmin}\sum _{i=1}^n{[{\{\widehat{m}({\mathit{H}}_{2i})-{\mu }_m({\mathit{H}}_{1i},A_{1i};{\mathit{\theta }}_m)\}}^2-{\sigma }_m^2({\mathit{H}}_{1i},A_{1i};{\mathit{\gamma }}_m)]}^2,$$

  and subsequently the estimator ${\sigma }_m^2({\mathit{H}}_1,A_1;{\widehat{\mathit{\gamma }}}_m)$ of ${\sigma }_m^2({\mathit{H}}_1,A_1)$. One choice for σ_m(h₁, a₂; γ_m) is a log-linear model, which may include non-linear basis terms.

• 3.3 Repeat Steps 3.1 and 3.2 to obtain estimators μ_c(H₁, A₁; θ̂_c) and σ_c(H₁, A₁; γ̂_c).

• 3.4 Compute standardized residuals ${\widehat{e}}_i^m$ and ${\widehat{e}}_i^c$, i = 1, …, n, as

  $$\begin{array}{cc}{\widehat{e}}_i^m=\frac{\widehat{m}({\mathit{H}}_{2i})-{\mu }_m({\mathit{H}}_{1i},A_{1i};{\widehat{\mathit{\theta }}}_m)}{{\sigma }_m({\mathit{H}}_{1i},A_{1i};{\widehat{\mathit{\gamma }}}_m)},& {\widehat{e}}_i^c=\frac{\widehat{c}({\mathit{H}}_{2i})-{\mu }_c({\mathit{H}}_{1i},A_{1i};{\widehat{\mathit{\theta }}}_c)}{{\sigma }_c({\mathit{H}}_{1i},A_{1i};{\widehat{\mathit{\gamma }}}_c)}.\end{array}$$

Then, ${\widehat{e}}_i^m$ and ${\widehat{e}}_i^c$, i = 1, …, n, can be used to estimate the joint distribution of the standardized residuals. Samples drawn from this distribution can be transformed back to samples from Ĝ(·, · | h₁, a₁) to estimate the integral I{y, F̂_ε(·), Ĝ(·, · | h₁, a₁)} with a Monte Carlo average.

3.3 Theoretical results

The following assumptions are used to establish consistency of the threshold exceedance probability and quantile that result from applying the estimated TIQ- and QIQ-learning optimal regimes, respectively. For each h₁, a₁, and h₂:

A1. the method used to estimate m(·) and c(·) results in estimators m̂(h₂) and ĉ(h₂) that converge in probability to m(h₂) and c(h₂), respectively;

A2. F_ε(·) is continuous, F̂_ε(·) is a cumulative distribution function, and F̂_ε(y) converges in probability to F_ε(y) uniformly in y;

A3. ∫|dĜ(u, v | h₁, a₁) – dG(u, v | h₁, a₁)| converges to zero in probability;

A4. $n^{-1}{\sum }_{i=1}^n\int |d\widehat{G}(u,v|{\mathit{H}}_{1i},a_1)-dG(u,v|{\mathit{H}}_{1i},a_1)|$ converges to zero in probability.

In the simulation experiments in Section 4 and data example in Section 5, we use linear working models for m(·) and c(·) that are estimated using least squares. Thus, A1 is satisfied under usual regularity conditions. When ε is continuous, assumption A2 can be satisfied by specifying F̂_ε(·) as the empirical distribution function. If for each fixed h₁ and a₁, dG(·, · | h₁, a₁) is a density and dĜ(·, · | h₁, a₁) a pointwise consistent estimator, then A3 is satisfied (Glick, 1974). Theorems 3.2 and 3.3 are proved in the supplemental material.

Theorem 3.2. (Consistency of TIQ-learning) Assume A1–A3 and fix λ ∈ ℝ. Then, ${\mathit{\text{pr}}}^{{\widehat{\mathit{\pi }}}_{\lambda }^{\mathit{\text{TIQ}}}}(Y>\lambda )$ converges in probability to ${\mathit{\text{pr}}}^{{\mathit{\pi }}_{\lambda }^{\mathit{\text{TIQ}}}}(Y>\lambda )$, where ${\widehat{\mathit{\pi }}}_{\lambda }^{\mathit{\text{TIQ}}}=({\widehat{\pi }}_{1,\lambda }^{\mathit{\text{TIQ}}},{\widehat{\pi }}_2^{\ast })$.

Theorem 3.3. (Consistency of QIQ-learning) Assume A1–A4. Then, $q^{{\widehat{\mathit{\pi }}}_{\tau }^{\mathit{\text{QIQ}}}}(\tau )$ converges in probability to $y_{\tau }^{\ast }$ for any fixed τ, where ${\widehat{\mathit{\pi }}}_{\tau }^{\mathit{\text{QIQ}}}=(\widehat{\mathrm{\Gamma }}(\cdot ,{\widehat{y}}_{\tau }^{\ast }),{\widehat{\pi }}_2^{\ast })$.

4. Simulation Experiments

We compare the performance of our estimators to binary Q-learning (Chakraborty and Moodie, 2013), Q-learning, and mean-optimal IQ-learning (Laber et al., 2014a) for a range of data generative models. Gains are achieved in terms of the proportion of the distribution of Y that exceeds the constant threshold λ and the τ^th quantile for several values of λ and τ. The data are generated using the model

$$\begin{array}{ccc}{\mathit{X}}_1~\text{Norm}({\mathbf{1}}_2,\sum ),& A_1,A_2~\text{Unif}{\{-1,1\}}^2,& {\mathit{H}}_1={(1,{\mathit{X}}_1^{\mathrm{T}})}^{\mathrm{T}},\\ {\eta }_{{\mathit{H}}_1,A_1}=exp\{\frac{C}{2}({\mathit{H}}_1^{\mathrm{T}}{\mathit{\gamma }}_0+A_1{\mathit{H}}_1^{\mathrm{T}}{\mathit{\gamma }}_1)\},& \mathit{\xi }~\text{Norm}({\mathbf{0}}_2,{\mathbf{I}}_2),& {\mathit{X}}_2={\mathit{B}}_{A_1}{\mathit{X}}_1+{\eta }_{{\mathit{H}}_1,A_1}\mathit{\xi },\\ {\mathit{H}}_2={(1,{\mathit{X}}_2^{\mathrm{T}})}^{\mathrm{T}},& \epsilon ~\text{Norm}(0,1),& Y={\mathit{H}}_2^{\mathrm{T}}{\mathit{\beta }}_{2,0}+A_2{\mathit{H}}_2^{\mathrm{T}}{\mathit{\beta }}_{2,1}+\epsilon ,\end{array}$$

where 1_p is a p × 1 vector of 1s, I_q is the q × q identify matrix, and C ∈ [0,1] is a constant. The matrix Σ is a correlation matrix with off-diagonal ρ = 0.5. The 2 × 2 matrix B_A1 equals

$$\begin{array}{cc}{\mathit{B}}_{A_1=1}=\left(\begin{array}{cc}-0.1& -0.1\\ 0.1& 0.1\end{array}\right),& {\mathit{B}}_{A_1=-1}=\left(\begin{array}{cc}0.5& -0.1\\ -0.1& 0.5\end{array}\right).\end{array}$$

The remaining parameters are γ₀ = (1, 0.5, 0)^T, γ₁ = (−1, −0.5, 0)^T, β_2,0 = (0.25, −1, 0.5)^T, and β_2,1 = (1, −0.5, −0.25)^T, which were chosen to ensure that the mean-optimal treatment produced a more variable response for some patients.

4.1 TIQ-learning Simulation Results

Results are based on J = 1,000 generated data sets. For each, we estimate the TIQ-, IQ-, binary Q-learning, and Q-learning policies using a training set of size n = 250 and compare the results using a test set of size N = 10, 000. The normal scale model is used to estimate F_ε(·), which is correctly specified for the generative model above. The Gaussian copula model discussed in Section 2.4 is also correctly specified and is used as the estimator for G(·, · | h₁, a₁). Results using a bivariate kernel estimator for G(·, · | h₁, a₁) are presented in the supplemental material.

To study the performance of the TIQ-learning algorithm, we compare values of the cumulative distribution function of the final response when treatment is assigned according to the estimated TIQ-learning, IQ-learning, binary Q-learning, and Q-learning regimes. Define pr^π̂_j(Y > λ) to be the true probability that Y exceeds λ given treatments are assigned according to π̂_j = (π̂_1j, π̂_2j), the regime estimated from the j^th generated data set. For threshold values λ = −2, 2, 4, we estimate pr^π(Y > λ) using ${\sum }_{j=1}^J{\widehat{\text{pr}}}^{{\widehat{\mathit{\pi }}}_j}(Y>\lambda )/J$, where pr̂^π̂j(Y > λ) is an estimate of pr^π̂j(Y > λ) obtained by calculating the proportion of test patients consistent with regime π̂_j whose observed Y values are greater than λ. Thus, our estimate is an average over training data sets and test set observations. In terms of the proportion of distribution mass above λ, results for λ = −2 and 4 in Figure 1 show a clear advantage of TIQ-learning for higher values of C, the degree of heteroskedasticity in the second-stage covariates X₂. As anticipated by Remark 2.3 in Section 2.1, all methods perform similarly when λ = 2.

Figure 1.

Left to Right: λ = −2, 2, 4. Solid black, true optimal threshold probabilities; dotted black, probabilites under randomization; dashed with circles/squares/crossed squares/triangles, probabilities under TIQ-, Q-, binary Q-, and Interactive Q-learning, respectively.

Figure 2 illustrates how the optimal first-stage treatment for a test set of 1,000 individuals changes as λ varies. Results are shown for C = 0.5. The true optimal treatments displayed in the left plot show a distinct shift from treating most of the population with A₁ = 1 to A₁ = −1 as λ increases from -4 to 4. The TIQ-learning estimated optimal treatments displayed in the middle plot are averaged over 100 Monte Carlo iterations and closely resemble the true policies on the left. Although the estimated Q-learning regime does not depend on λ, it is plotted for each λ value to aid visual comparison. The first-stage treatments recommended by Q-learning differ the most from the true optimal treatments when λ = 4, corroborating the results for C = 0.5 in Figure 1. The rightmost panel of Figure 2 are the results from binary Q-learning with the binary outcome defined as 𝟙_Y>λ. While there appears to be a slight deviation in the results from mean-optimal Q-learning for λ values 3 and 4, overall the resulting policies are similar to mean-optimal Q-learning and do not recover the true optimal treatments on average.

Figure 2.

From left: True optimal first-stage treatments for 1,000 test set patients when λ = −4, −3, …, 4, coded light gray when ${\pi }_{1,\lambda }^{\text{TIQ}}(h_1)=1$ and dark gray otherwise; TIQ-learning estimated optimal first-stage treatments; Q-learning estimated optimal first-stage treatments, plotted constant in λ to aid visual comparison; and binary Q-learning estimated optimal first-stage treatments for each λ.

4.2 QIQ-learning Simulations

To study the performance of the QIQ-learning algorithm, we compare quantiles of Y when the population is treated according to the regimes estimated by QIQ-learning, IQ-learning, and Q-learning. A smaller test set of size N = 5,000 was used in this section to reduce computation time. Define q^π̂_j (τ) to be the true τ^th quantile of the distribution of Y given treatments are assigned according to π̂_j = (π̂_1j, π̂_2j), the regime estimated from the j^th generated data set. For τ = 0.1, 0.5, 0.75, we estimate q^π(τ) using ${\sum }_{j=1}^J{\widehat{q}}^{{\widehat{\mathit{\pi }}}_j}(\tau )/J$, where q̂^π̂_j(τ) is an estimate of q^π̂_j(τ) obtained by calculating the τ^th quantile of the subgroup of test patients consistent with regime π̂_j. The generative model and all other parameter settings used here are the same as those in the previous section. For our generative model, the condition of Lemma 3.1 is satisfied, so the true optimal regime is attained asymptotically. The results in Figure 3 indicate that the lowest quantile, τ = 0.1, suffers under the Q-learning regime as heterogeneity in the second-stage histories increases, measured by the scaling constant C. In contrast, quantiles of the QIQ-learning estimated regimes for τ = 0.1 remain constant across the entire range of C. When τ = 0.5, all methods perform similarly; for some C, IQ- and Q-learning outperform QIQ-learning. This is not surprising because all models used to generate the data were symmetric. Thus, maximizing the mean of Y gives similar results to maximizing the median.

Figure 3.

Left to Right: τ = 0.1, 0.5, 0.75. Solid black, true optimal quantiles; dotten black, quantiles under randomization; dashed with circles/squares/triangles, quantiles under QIQ-, Q-, and IQ-learning, respectively.

Next we study QIQ-learning when the first stage errors are skewed. The generative model and parameter settings used here are the same as those used previously except that

$$\begin{array}{cc}{\mathit{X}}_2={\mathit{B}}_{A_1}{\mathit{X}}_1+{\eta }_{{\mathit{H}}_1,A_1}\mathit{\xi },& {\eta }_{{\mathit{H}}_1,A_1}=exp\{\frac{1}{2}({\mathit{H}}_1^{\mathrm{T}}{\mathit{\gamma }}_0+A_1{\mathit{H}}_1^{\mathrm{T}}{\mathit{\gamma }}_1)\},\\ (10C+1)+\sqrt{2(10C+1)}\mathit{\xi }~{\chi }_{df=10C+1}^2,\end{array}$$

where C ∈ [0, 1] is a constant that reflects the degree of skewness in the first-stage errors, ξ. Smaller values of C correspond to heavier skew.

Results are averaged over J = 100 generated data sets; for each, we estimate the QIQ-, IQ-, and Q-learning policies and compare the results using a test set of size N = 10, 000. The training sample size for each iteration is n = 500. The normal scale model is used to estimate F_ε(·), which is correctly specified. A bivariate kernel density estimator is used to estimate G(·, · | h₁, a₁). As before, we compare quantiles of the final response when treatment is assigned according to the estimated QIQ-learning, IQ-learning, and Q-learning regimes, and the results are given in Figure 4. QIQ-learning demonstrates an advantage over the mean-optimal methods for all three quantiles and almost uniformly across the degree of skewness of the first-stage errors.

Figure 4.

Left to Right: τ = 0.1,0.5,0.75. Solid black, true optimal threshold probabilities; dotted black, probabilites under randomization; dashed with circles/squares/triangles, probabilities under TIQ-, Q-, and Interactive Q-learning, respectively. Training set size of n = 500.

5. Star*D Analysis

The Sequenced Treatment Alternatives to Relieve Depression (STAR*D) trial (Fava et al., 2003; Rush et al., 2004) is a four-stage Sequential Multiple Assignment Randomized Trial (Lavori and Dawson, 2004; Murphy, 2005a) studying personalized treatment strategies for patients with major depressive disorder. Depression is measured by the Quick Inventory of Depressive Symptomatology (QIDS) score, a one-number summary score that takes integer values 0 to 27. Lower scores indicate fewer depression symptoms. Remission is defined as QIDS ≤ 5. Previous attempts to estimate optimal dynamic treatment regimes from this data have used the criteria, “maximize end-of-stage-two QIDS,” (see, for example, Schulte et al., 2012; Laber et al., 2014a) a surrogate for the primary aim of helping patients achieve remission. We illustrate TIQ-learning by estimating an optimal regime that maximizes the probability of remission for each patient, directly corresponding to the primary clinical goal.

The first stage, which we will henceforth refer to as baseline, was non-randomized with each patient receiving Citalopram, a drug in the class of Selective Serotonin Reuptake Inhibitors (SSRIs). We use a subset of the STAR*D data from the first two randomized stages, and refer to the original trial levels 2 and 3 as “stage one” and “stage two.” Before each randomization, patients specified a preference to “switch” or “augment” their current treatment strategy and were then randomized to one of multiple options within their preferred category. In addition, patients who achieved remission in any stage exited the study. To keep our illustration of TIQ-learning concise, we restrict our analysis to the subset of patients who who preferred the “switch” strategy at both stages. We note that this subgroup is not identifiable at baseline because patient preferences depend on the assigned treatment and subsequent response at each stage. Our motivation for this restriction is to mimic a two-stage SMART where treatments are randomized at both stages, thus simplifying our illustration. At stage one, our binary treatment variable is “SSRI,” which includes only Sertraline, versus “non-SSRI,” which includes both Bupropion and Venlafaxine. At stage two we compare Mirtazapine and Nortriptyline which are both non-SSRIs. In the patient subgroup considered in our analysis, treatments were randomized at both stages.

All measured QIDS scores are recoded as 27– QIDS so that higher scores correspond to fewer depression symptoms. After recoding, remission corresponds to QIDS > 21. Thus, TIQ-learning with λ = 21 maximizes the probability of remission for all patients. In general, QIDS was recorded during clinic visits at weeks 2, 4, 6, 9, and 12 in each stage, although some patients with inadequate response moved on to the next stage before completing all visits. We summarize longitudinal QIDS trajectories from the baseline stage and stage one by averaging over the total number of QIDS observations in the given stage. Variables used in our analysis are listed in Table 1. We describe all models used in the analysis below.

Table 1.

Variables used in the STAR*D analysis.

Variable	Description
qids0	mean QIDS during the baseline stage.
slope0	pre-randomization QIDS improvement; the difference between the final and initial baseline-stage QIDS scores, divided by time spent in the baseline stage.
qids1	mean stage-one QIDS.
slope1	first-stage QIDS improvement; the difference between the final and initial first-stage QIDS scores, divided by time spent in the first randomized stage.
A1	First-stage treatment; 1=“SSRI” and -1=“non-SSRI.”
A2	Second-stage treatment; 1=“NTP” for Nortriptyline and -1=“MIRT” for Mirtazapine.
Y	27 minus final QIDS score, measured at the end of stage two.

At the second stage, we assume the linear working model $Y={\mathit{H}}_{2,0}^{\mathrm{T}}{\mathit{\beta }}_{2,0}+A_2{\mathit{H}}_{2,1}^{\mathrm{T}}{\mathit{\beta }}_{2,1}+\epsilon$, where H_2,0 = H_2,1 = (1, qids1, slope1, A1)^T, E(ε) = 0, var(ε) = σ², and ε is independent of H₂ and A₂. We fit this model using least squares. A normal qq-plot of the residuals from the previous regression step indicates slight deviation from normality, so we use the non-parametric estimator of F_ε(·) given in Section 2.4. Next, we estimate the conditional mean and variance functions of $m({\mathit{H}}_2)\triangleq {\mathit{H}}_{2,0}^{\mathrm{T}}{\mathit{\beta }}_{2,0}$ and $c({\mathit{H}}_2)\triangleq {\mathit{H}}_{2,1}^{\mathrm{T}}{\mathit{\beta }}_{2,1}$ following steps described in Section 2.4. For the mean functions, we take ${\mathit{H}}_{1,0}={\mathit{H}}_{1,1}={(1,{\mathit{X}}_1^{\mathrm{T}})}^{\mathrm{T}}$ with X₁ = (qids0, slope0)¹ and use working models of the form $E\{k({\mathit{H}}_2)|{\mathit{X}}_1{A}_1\}={\mathit{H}}_{1,0}^{\mathrm{T}}{\mathit{\beta }}_{1,0}^k+A_1{\mathit{H}}_{1,1}^{\mathrm{T}}{\beta }_{1,1}^k$. Exploratory analyses reveal little evidence of heteroskedasticity at the first-stage. Thus, we opt to estimate a constant residual variance for both terms following the mean modeling steps. After the mean and variance modeling steps, we use a Gaussian copula to estimate the joint conditional distribution of the standardized residuals of {m(H₂), c(H₂)} given H₁ and A₁, resulting in our estimate of G(·, · | h₁, a₁) which we denote by Ĝ(·, · | h₁, a₁).

The estimated first-stage optimal rule is ${\widehat{\pi }}_{1,\lambda }^{\text{TIQ}}({\mathit{h}}_1)=\underset{a_1}{argmin}\int {\widehat{F}}_{\epsilon }(21-u-|v|)d\widehat{G}(u,v|{\mathit{h}}_1,a_1)$. At stage two, ${\widehat{\pi }}_2^{\ast }({\mathit{h}}_2)=sgn(-1.66+0.15\ast \text{qids}1-4.03\ast \text{slope}1-0.68\ast \mathrm{A}1)$ is the estimated optimal treatment. Based on Remark 1 in Section 2.1, we compare the estimated first-stage treatment recommendations to those recommended by the mean-optimal rule, arg min_a1 ∫(−u – |v|)dĜ(u, v | h₁, a₁), for each observed h₁ in the data. Only one patient out of 132 is recommended differently. In addition, the difference in raw values of ∫F̂_ε(21 – u – |v|)dĜ(u, v | h₁, a₁) for a₁ = 1, −1 as well as ∫(−u – |v|)dĜ(u, v | h₁, a₁) for a₁ = 1, −1 are the smallest for this particular patient. Thus, the treatment discrepancy is most likely due to a near-zero treatment effect for this patient.

We compare TIQ-learning to the Q-learning analysis of Schulte et al. (2012) and binary Q-learning (Chakraborty and Moodie, 2013). Comparing the results to Q-learning, which maximizes the expected value of Y, supports the claim that TIQ-learning and mean optimization are equivalent for this subset of the STAR*D data. The first step of Q-learning is to model the conditional expectation of Y given H₂ and A₂ which is the same as the first step of TIQ-learning. Thus, we use the same model and estimated decision rule at stage two given in Step 1 of the TIQ-learning algorithm. Next, we model the conditional expectation of $\stackrel{\sim }{Y}={\mathit{H}}_{2,0}^{\mathrm{T}}{\mathit{\beta }}_{2,0}+|{\mathit{H}}_{2,1}^{\mathrm{T}}{\beta }_{2,1}|$, where Ỹ is the predicted future optimal outcome at stage one. We specify the working model $E(\stackrel{\sim }{Y}|{\mathit{H}}_1,A_1)={\mathit{H}}_{1,0}^{\mathrm{T}}{\mathit{\beta }}_{1,0}^Q+A_1{\mathit{H}}_{1,1}^{\mathrm{T}}{\mathit{\beta }}_{1,1}^Q$, where ${\mathit{H}}_{1,0}^{\mathrm{T}}={\mathit{H}}_{1,1}^{\mathrm{T}}={(1,{\mathit{X}}_1^{\mathrm{T}})}^{\mathrm{T}}$ and X₁ = (qids0, slope0)^T. We fit the model using least squares. Then, the Q-learning estimated optimal first-stage rule is ${\widehat{\pi }}_{1,\lambda }^Q({\mathit{h}}_1)=sgn(-0.95+0.13\ast \text{qids}1-2.17\ast \text{slope}1)$. Q-learning recommends treatment differently at the first stage for only one of the 132 patients in the data. In addition, the estimated value of the TIQ- and Q-learning regimes are nearly the same and are displayed in Table 2. Binary Q-learning recommends treatment differently than TIQ-learning for 18 patients at the first stage, and the estimated value of the binary Q-learning regime is slightly lower than TIQ- and Q-learning. Also included in Table 2 are value estimates for four non-dynamic regimes that treat everyone according to the decision rules π₁(h₁) = a₁ and π₂(h₂) = a₂ for a₁ ∈ {−1, 1} and a₂ ∈ {−1, 1}. We estimate these values using the Augmented Inverse Probability Weighted Estimator given in Zhang et al. (2013).

Table 2.

Estimated value of dynamic and non-dynamic regimes using the Adaptive Inverse Probability Weighted Estimator.

	Estimated Value
TIQ-learning	0.24
Q-learning	0.23
Binary Q-learning	0.19
(1, 1)	0.13
(-1, 1)	0.24
(1, -1)	0.07
(-1, -1)	0.12

In summary, it appears that TIQ-learning and Q-learning perform similarly for this subset of the STAR*D data. This may be due to the lack of heteroskedasticity at the first stage. Thus, maximizing the end-of-stage-two QIDS using mean-optimal techniques seems appropriate and, in practice, equivalent to maximizing remission probabilities for each patient with TIQ-learning.

6. Discussion

We have proposed modeling frameworks for estimating optimal dynamic treatment regimes in settings where a non-mean distributional summary is the intended outcome to optimize. Threshold Interactive Q-learning (TIQ-learning) estimates a regime that maximizes the mass of the response distribution that exceeds a constant or patient-dependent threshold. Although TIQ-learning is just a special case of optimizing regimes based on binary responses, it is a helpful precursor for the development of Quantile Interactive Q-learning (QIQ-learning), which maximizes a prespecified quantile of the response distribution. If the focus is simply on optimizing a dynamic treatment regime for a binary threshold outcome, rather than quantile optimization, recent nonparametric techniques tailored to the binary outcome setting may be preferred to avoid problems due to model misspecification (Zhao et al., 2012, 2015). For example, it is possible to prespecify a class of regimes and directly optimize within that class by utilizing the value function,

$$V^{{\pi }_1,{\pi }_2}=\frac{{\sum }_{i=1}^n{\mathbb{1}}_{Y_i>\lambda }{\mathbb{1}}_{A_{1,i}={\pi }_1({\mathit{H}}_{1,i})}{\mathbb{1}}_{A_{2,i}={\pi }_2({\mathit{H}}_{2,i})}}{{\sum }_{i=1}^n{\mathbb{1}}_{A_{1,i}={\pi }_1({\mathit{H}}_{1,i})}{\mathbb{1}}_{A_{2,i}={\pi }_2({\mathit{H}}_{2,i})}}.$$

To our knowledge, this is the first attempt to estimate an optimal treatment regime that targets a quantile. Despite generalizations presented in the supplementary material, some researchers might be wary of the assumptions needed for consistency of the estimated optimal regime or the requisite modeling of the main effect function. An interesting future direction would be to develop a flexible framework for QIQ-learning that depends only on models for the treatment contrast function and not on the main effect term. In addition, it has been shown that Q-learning, A-learning, and g-estimation lead to identical estimators in certain cases but that Q-learning is less efficient (Chakraborty et al., 2010; Schulte et al., 2012). It is possible a g-estimation approach exists for maximizing quantiles based on structural nested distribution models (Robins, 2000; Vansteelandt et al., 2014).

Our proposed methods are designed for the two-stage setting, this is an important development given that many completed and ongoing SMART studies have this structure (Projects Using SMART, 2012; Laber, 2013). Here we considered binary treatments at both stages. In principle, the proposed methods can be extended to settings with more than two treatments at each stage by modeling additional treatment contrasts. Formalization of this idea merits further research.

7. Supplementary Materials

Online supplementary materials include discussions of modeling adjustments for heteroskedastic second-stage errors and patient-specific thresholds, a proof of Lemma 3.1 and toy example illustrating where this lemma does not apply, additional simulation results, and proofs of the theorems in Section 3.3.