Interactive model building for Q-learning

Summary

Evidence-based rules for optimal treatment allocation are key components in the quest for efficient, effective health care delivery. Q-learning, an approximate dynamic programming algorithm, is a popular method for estimating optimal sequential decision rules from data. Q-learning requires the modeling of nonsmooth, nonmonotone transformations of the data, complicating the search for adequately expressive, yet parsimonious, statistical models. The default Q-learning working model is multiple linear regression, which is not only provably misspecified under most data-generating models, but also results in nonregular regression estimators, complicating inference. We propose an alternative strategy for estimating optimal sequential decision rules for which the requisite statistical modeling does not depend on nonsmooth, nonmonotone transformed data, does not result in nonregular regression estimators, is consistent under a broader array of data-generation models than Q-learning, results in estimated sequential decision rules that have better sampling properties, and is amenable to established statistical approaches for exploratory data analysis, model building, and validation. We derive the new method, IQ-learning, via an interchange in the order of certain steps in Q-learning. In simulated experiments IQ-learning improves on Q-learning in terms of integrated mean squared error and power. The method is illustrated using data from a study of major depressive disorder.

1. Introduction

Clinical treatment decisions are based on a patient’s treatment history and current health status. Sequential decision rules, also known as dynamic treatment regimes, formalize this process by specifying a sequence of decision rules, one for each treatment decision, that take as input a patient’s treatment and covariate history and output recommended treatments. An optimal sequential decision rule is one that maximizes a desirable clinical outcome. The sequential nature of clinical decision making problems has led researchers to estimate optimal sequential decision rules using approximate dynamic programming procedures (Robins, 2004; Murphy, 2005a). Q-learning with function approximation, hereafter Q-learning, is one such popular method (Murphy, 2005a). However, Q-learning and similar variants of it involve modeling nonsmooth, nonmonotone functions of the data. Nonmonotonicity complicates the regression function whereas nonsmoothness imparts nonregularity to estimators. Inference in the presence of such nonregularity has been well-studied by Robins (2004), Chakraborty et al. (2010), Laber et al. (2014), and Moodie and Richardson (2010). However, much less attention has been directed toward the effect of nonsmoothness on the equally-important applied problems of model building and diagnostics. The preceding references all study linear working models, making little mention of their appropriateness or how to interactively build a model using data. As we demonstrate, even under simple generative models, linear working models result in questionable fits.

Rather than develop specialized exploratory and model building techniques for Q-learning, we propose to model the data before applying the necessary nonmonotone, nonsmooth operations. Because standard interactive model building techniques can be used with our new version of Q-learning, we call it IQ-learning for interactive Q-learning. Interactive model building is an essential part of extracting meaningful information from data (Henderson and Velleman, 1981; Cook and Weisberg, 1982; Henderson et al., 2010; Rich et al., 2010; Chakraborty and Moodie, 2013). This is especially true when the analysis is intended to inform clinical practice or provide scientific insight, and thus IQ-learning is especially attractive in applications.

IQ-learning has several advantages over Q-learning. For a large class of generative models, IQ-learning involves only simple, well-studied, and well-understood conditional mean and variance modeling of smooth transformations of the data, resulting in better fitting and more interpretable models (Carroll and Ruppert, 1988). Furthermore, inference for coefficients indexing the working models in IQ-learning is greatly simplified by regular normal limit theory. However, IQ-learning does not resolve the problem of nonregularity of the estimated non-terminal Q-functions, just of the coefficients on which they depend. Nevertheless, this is an important distinction from Q-learning and related methods. This issue is discussed further in Section 2.3.

2. Q- and IQ-Learning

2·1. Setup

For simplicity we consider the case of two treatment decisions and two treatment options per stage. In addition to being the most common in practice (see www4.stat.ncsu.edu/~laber/smarts.html), the two-stage, binary-treatment setting facilitates exposition while maintaining the key features of more general problems. While we focus on randomized studies, the proposed approach is valid for observational data under the same causal conditions required for Q-learning (Schulte et al., 2012).

Training data for the two-stage, two-treatment case, $𝒟={\{(X_{1,i},A_{1,i},X_{2,i},A_{2,i},Y_i)\}}_{i=1}^n$, consists of independent identically distributed copies of the quintuple (X₁, A₁, X₂, A₂, Y) containing data collected on a single subject. Each quintuple is time ordered and thus called a trajectory: X₁ ∈ ℝ^p₁ is baseline covariate information; A₁ ∈ {−1, 1} is the first treatment; X₂ ∈ ℝ^p₂ is covariate information collected between the first and second treatment assignments; A₂ ∈ {−1, 1} is the second treatment; and Y ∈ ℝ is the outcome, coded so that higher values coincide with more desirable clinical outcomes. For notational compactness and conformity with established practice, we denote the information available prior to the t^th treatment assignment by H_t. Thus, H₁ = X₁ and $H_2={(X_1^{\top },A_1,X_2^{\top })}^{\top }$.

2·2. Q-Learning

The goal of Q- and IQ-learning is estimation of a pair of decision rules π = (π₁, π₂) such that π _t maps the domain of H_t into the set of treatments, π_t : ℋ_t ↦ {−1, 1}. An optimal sequential decision rule π^opt maximizes expected outcome. Let E^π denote expectation under the restriction A_t = π_t(H_t); then π^opt satisfies E^πopt (Y) = sup_π E^π (Y).

Q-learning, a regression-based, approximate dynamic programming algorithm, is commonly used to estimate π^opt (Murphy, 2005a). The algorithm depends on the Q-functions,

$$Q_2(h_2,a_2)=E(Y|H_2=h_2,A_2=a_2),$$ (1)

$$Q_1(h_1,a_1)=E\{\underset{a_2\in \{-1,1\}}{\text{max}}{Q}_2(H_2,a_2)|H_1=h_1,A_1=a_1\}.$$ (2)

Thus, Q_t(h_t, a_t) measures the quality of treatment a_t when assigned to a patient with history h_t, assuming that optimal treatment decisions are made in future stages (Sutton and Barto, 1998). If the Q-functions were known, the optimal sequential decision rule could be determined using dynamic programming (Bellman, 1957), yielding the solution ${\pi }_t^{\text{opt}}(h_t)={\text{arg max}}_{a_t\in \{-1,1\}}{Q}_t(h_t,a_t)(t=1,2)$. The Q-functions are seldom known, and in practice Q-learning mimics the dynamic programming solution by replacing the unknown conditional expectations with fitted regression models. For reasons of data scarcity, model parsimony, and simplicity, it is common to use linear models for the Q-functions:

$$Q_t(h_t,a_t;{\beta }_t)=h_{t,0}^{\top }{\beta }_{t,0}+a_t{h}_{t,1}^{\top }{\beta }_{t,1},t=1,2,$$ (3)

where h_t,0 and h_t,1 are, possibly the same, subvectors of h_t, and ${\beta }_t={({\beta }_{t,0}^{\top },{\beta }_{t,1}^{\top })}^{\top }$. Q-learning with linear models consists of three steps:

• Q1estimate β₂, and hence Q₂, via least squares regression of Y on H₂ and A₂ using the working model (3), i.e., ${\mathrm{\beta ̂}}_2={\text{arg min}}_{{\beta }_2}{\sum }_{i=1}^n{\{Y_i-Q_2(H_{2,i},A_{2,i};{\beta }_2)\}}^2$;
• Q2
  calculate predicted future outcomes Ỹ assuming optimal Stage 2 decisions

  $$Ỹ={\text{max}}_{a_2\in \{-1,1\}}{Q}_2(H_2,a_2;{\mathrm{\beta ̂}}_2)=H_{2,0}^{\top }{\mathrm{\beta ̂}}_{2,0}+|H_{2,1}^{\top }{\mathrm{\beta ̂}}_{2,1}|;$$

  then estimate β₁, and hence Q₁, via least squares regression of Ỹ on H₁ and A₁ using the working model (3), i.e., ${\mathrm{\beta ̂}}_1={\text{arg min}}_{{\beta }_1}{\sum }_{i=1}^n{\{{Ỹ}_i-Q_1(H_{1,i},A_{1,i};{\beta }_1)\}}^2$; and
• Q3
  calculate the estimated Q-learning optimal treatment policy ${\mathrm{\pi ̂}}^Q=({\mathrm{\pi ̂}}_1^Q,{\mathrm{\pi ̂}}_2^Q)$ as

  $${\mathrm{\pi ̂}}_t^Q(h_t)={\text{arg max}}_{a_t\in \{-1,1\}}{Q}_t(h_t,a_t;{\mathrm{\beta ̂}}_t).$$

There is nothing unusual about the regression modeling in the first step, which is amenable to well-studied techniques and diagnostics. The same is not true of the modeling in the second step. The absolute value function in the definition of Ỹ means that even if the relationships among the components of H₁ and H₂ are approximately linear, the dependence of Ỹ on H₁ would necessarily be nonmonotone. Thus the working model for Ỹ commonly used in practice is generally wrong. Furthermore, there are no simple transformations of the observed variables or simple modifications to the working model that render the true regression of Ỹ on H₁ and A₁ linear, even asymptotically. We illustrate this point with data generated from the model

$$\begin{array}{cc}{X}_1~\mathrm{N}(0,{\sigma }^2),\hfill & \xi ~\mathrm{N}(0,{\tau }^2),X_2=\zeta X_1+\xi ,\hfill \\ A_t~\text{Unif}\{-1,1\},t=1,2,\hfill & \varphi ~\mathrm{N}(0,{\gamma }^2),Y=1.25A_1{A}_2+A_2{X}_2-A_1{X}_1+\varphi ,\hfill \end{array}$$ (4)

where σ², τ², ζ, and γ² are fixed parameters. In most applications one expects ζ = cov(X₁, X₂) ≠ 0. Treatments are randomly assigned at each stage as in a sequential multiple assignment randomized trial design (Murphy, 2005b; see also Lavori and Dawson, 2000, 2004). The working model in (3) for Q₂(H₂, A₂) is correct, so the resulting predicted value Ỹ approximates the true fitted value Ỹ_True = |1.25A₁ + X₂| − A₁X₁. It follows that the regression in Step Q2 approximates the regression of Ỹ_True on H₁ = X₁ and A₁. Substituting for X₂ in Ỹ shows that Ỹ_True = |1.25A₁ + ζX₁ + ξ| − A₁X₁, from which it is apparent that E(Ỹ_True|X₁, A₁) is linear in X₁ for fixed A₁ only in the unlikely case that ζ = 0. Thus correlation between X₁ and X₂ induces a nonlinear dependence of Ỹ on X₁.

The left-hand panel of Figure 1 displays a scatterplot of Ỹ against X₁ for each value of A₁ based on 1,000 random draws from model (4) with ζ = 0.85, σ = 1, and τ = γ = 1/√2, using the Q-learning algorithm to calculate Ỹ. The figure illustrates nonlinearity in the regression of Ỹ on X₁ and also heteroscedastic variation induced by the max operation in Step Q2. As this toy model makes clear, identifying the correct form of the regression model for E(Ỹ_True | H₁,A₁) and fitting it efficiently would be difficult in the realistic case that the data-generating model is unknown; one approach would be to adopt nonparametric models for the Q-functions (e.g., Zhao et al., 2011; Moodie et al., 2013), but some clinicians are wary of black-box approaches and it 125 can be difficult to glean scientific knowledge from these models. Thus, common practice is to ignore the problem and settle for the best approximation afforded by fitting linear models. This problem is shared by variants of Q-learning (e.g., A-learning, Murphy, 2003; Blatt et al., 2004; Robins, 2004; Schulte et al., 2012). In contrast, the right-hand panel of Fig. 1 shows the first-stage regression model that must be fit in our proposed method, IQ-learning, described next. It is a common analysis of covariance model in X₁ and A₁.

Fig. 1.

Scatterplots of Ỹ (left) and Δ̂(H₂) (right) against X₁ for A₁ = −1 (black circles) and A₁ = 1 (grey crosses) for 1, 000 random samples from the toy model. Step 2 of the Q-learning algorithm requires modeling the data in the left plot; note the nonlinearity and heteroscedasticity. Data in the right plot must be modeled for IQ-learning; note the common analysis of covariance structure.

2·3. IQ-Learning

IQ-learning replaces the difficult problem of modeling the predicted future optimal outcomes $Ỹ={\text{max}}_{a_2\in \{-1,1\}}{Q}_2(H_2,a_2;{\mathrm{\beta ̂}}_2)=H_{2,0}^{\top }{\mathrm{\beta ̂}}_{2,0}+|H_{2,1}^{\top }{\mathrm{\beta ̂}}_{2,1}|$ with two ordinary mean-variance function modeling problems. Its practical advantages result from the fact that there is a wealth of models and theory for mean-variance function modeling (Carroll and Ruppert, 1988). Thus it has the potential for better model building and diagnostics. The modeling required is familiar and is generally interactive. We first describe the IQ-learning algorithm in general terms and then discuss special cases that are useful in practice.

Whereas Q-learning models max_{a2∈{−1,1}} Q₂(H₂, a₂) directly, IQ-learning starts with the Q₂ contrast and main-effect functions: Δ(H₂) = {Q₂(H₂, 1) − Q₂(H₂, −1)} /2; μ(H₂) = {Q₂(H₂, 1) + Q₂(H₂, −1)} /2. The contrast and main-effect functions are linear, and hence smooth and monotone functions of Q₂(·). Let g_{h1, a1}(·) denote the conditional distribution of the contrast Δ(H₂) given H₁ = h₁ and A₁ = a₁. With these definitions Q₁(h₁, a₁) defined in (2) can be written as

$$Q_1(h_1,a_1)=E\{\mu (H_2)|H_1=h_1,A_1=a_1\}+\int |z|g_{h_1,a_1}(z)dz.$$ (5)

The IQ-learning estimator of Q₁(h₁, a₁) has the form

$${\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q}}(h_1,a_1)=\mathrm{L̂}(h_1,a_1)+\int |z|{ĝ}_{h_1,a_1}(z)dz,$$ (6)

where L̂(h₁, a₁) and ĝ_{h1, a1}(·) are estimators of E {μ(H₂) | H₁ = h₁, A₁ = a₁} and g_{h1, a1}(·).

Let Q̂₂(H₂, A₂) denote the estimator obtained in Step Q1 of the Q-learning algorithm. Define the estimated main-effect and contrast functions

$$\mathrm{\mu ̂}(H_2)=\frac{1}{2}\{{\mathrm{Q̂}}_2(H_2,1)+{\mathrm{Q̂}}_2(H_2,-1)\},\mathrm{\Delta ̂}(H_2)=\frac{1}{2}\{{\mathrm{Q̂}}_2(H_2,1)-{\mathrm{Q̂}}_2(H_2,-1)\}.$$ (7)

Then L̂(h₁, a₁) is obtained by modeling the regression of μ̂(H₂) on H₁ and A₁ for which linear models are often adequate as no unusual transformations are involved. Obtaining ĝ_{h1, a1}(·) is accomplished by estimating the conditional distribution of Δ̂(H₂) given H₁ and A₁. For this we exploit mean-variance function modeling as explained in Section 2·4. Thus we have the following algorithm for IQ-learning:

• IQ1use Step Q1 of the Q-learning algorithm to obtain β̂₂ and ${\mathrm{Q̂}}_2^{\mathrm{I}\mathrm{Q}}(H_2,A_2)=Q_2(H_2,A_2;{\mathrm{\beta ̂}}_2)$;
• IQ2

  1. regress the estimated main-effect function μ̂(H₂) from (7) on H₁ and A₁ to obtain an estimator L̂(h₁, a₁) of E {μ(H₂) | H₁ = h₁, a₁};

  2. model the conditional distribution of the estimated contrast function Δ̂(H₂) from (7) given H₁ = h₁ and A₁ = a₁ to obtain an estimator ĝ_{h1, a1}(·) of g_{h1, a1}(·);

  3. combine the estimators from IQ2a and IQ2b to obtain

     $${\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q}}(h_1,a_1)=\mathrm{L̂}(h_1,a_1)+\int |z|{ĝ}_{h_1,a_1}(z)dz;$$
• IQ3define the IQ-learning estimated optimal treatment policy ${\mathrm{\pi ̂}}^{\mathrm{I}\mathrm{Q}}=({\mathrm{\pi ̂}}_1^{\mathrm{I}\mathrm{Q}},{\mathrm{\pi ̂}}_2^{\mathrm{I}\mathrm{Q}})$ so that ${\mathrm{\pi ̂}}_t^{\mathrm{I}\mathrm{Q}}(h_t)={\text{arg max}}_{a_t\in \{-1,1\}}{\mathrm{Q̂}}_t^{\mathrm{I}\mathrm{Q}}(h_t,a_t)$.

Completing our algorithm requires specific models for Steps IQ1, IQ2a, and IQ2b. As noted previously, Steps IQ1 and IQ2a are usually straightforward and linear models will often suffice. We now show how to accomplish the modeling in Step IQ2b efficiently and with sufficient flexibility for many applications by using mean-variance models.

2·4. Location-Scale Working Models for g_{h1, a1}(·)

Henceforth we consider mean-variance, location-scale estimators of g_{h1, a1}(·) of the form

$${ĝ}_{h_1,a_1}(z)=\frac{1}{\mathrm{\sigma ̂}(h_1,a_1)}\mathrm{\varphi ̂}\left\{\frac{z-\mathrm{m̂}(h_1,a_1)}{\mathrm{\sigma ̂}(h_1,a_1)}\right\},$$ (8)

where m̂(h₁, a₁) is an estimator of m(h₁, a₁) = E {Δ(H₂) | H₁ = h₁, A₁ = a₁}, σ̂²(h₁, a₁) is an estimator of σ²(h₁, a₁) = E [{Δ(H₂) − m(h₁, a₁)}² | H₁ = h₁, A₁ = a₁], and ϕ̂ is an estimator of the density of the standardized residuals {Δ(H₂) − m(h₁, a₁)} /σ(h₁, a₁), say ϕ_{h1, a1}, which we assume does not depend on the history h₁ or the treatment a₁. In other words, we assume that all of the dependence of Δ(H₂) on (H₁, A₁) is captured by the conditional mean and variance functions. The great success of mean-variance function modeling (Carroll and Ruppert, 1988) suggests that this assumption is reasonable quite generally; however, we also note that substantial departures from the assumption can be investigated by stratifying on h₁ and a₁ and comparing higher-order moments, such as skewness and kurtosis, or nonparametric density estimates of the empirical residuals {Δ̂(H₂) − m̂(H₁,A₁)} /σ̂(H₁, A₁) across the strata. We now describe two special cases of the estimator in (8).

Let ϕ denote the density of a standard normal random variable. A simple but useful estimator of g_{h1, a1}(·) is the normal location-scale model:

$${ĝ}_{h_1,a_1}^N(z)=\frac{1}{\mathrm{\sigma ̂}(h_1,a_1)}\varphi \left\{\frac{z-\mathrm{m̂}(h_1,a_1)}{\mathrm{\sigma ̂}(h_1,a_1)}\right\},$$ (9)

which is a special case of (8) with ϕ̂ = ϕ. An advantage of this model is that $\int |z|{ĝ}_{h_1,a_1}^N(z)dz$ can be evaluated in closed form. In particular,

$$\int |z|{ĝ}_{h_1,a_1}^N(z)dz=\mathrm{m̂}(h_1,a_1)\left[2\mathrm{\Phi }\right\{\frac{\mathrm{m̂}(h_1,a_1)}{\mathrm{\sigma ̂}(h_1,a_1)}\}-1]+2\mathrm{\sigma ̂}(h_1,a_1)\varphi \left\{\frac{\mathrm{m̂}(h_1,a_1)}{\mathrm{\sigma ̂}(h_1,a_1)}\right\},$$ (10)

where Φ is the standard normal cumulative distribution function. If the mean and variance functions are correctly specified and ϕ_{h1, a1} = ϕ, then the IQ-learning location-scale model is correct. As commonly implemented, Q-learning fits a misspecified model and thus estimators are not consistent. The closed form expression in (10) makes it possible to study the bias in Q-learning when the true data-generating model is a normal mean-variance function model. Details are in the Supplementary Material.

The normal location-scale model assumes that ϕ_{h1, a1} = ϕ, the standard normal density. Violations of normality can be diagnosed via examination of the observed standardized residuals, ê_i = {Δ̂(H_2,i) − m̂{(H_{1, i}, A_{1, i})}/σ̂(H_1,i, A_1,i) (i = 1, …, n). When greater modeling flexibility is desired, the normality assumption can be dropped and the empirical distribution of the ê_i used instead. Defining

$${Ĝ}_{h_1,a_1}(z)=\frac{1}{n}\sum _{i=1}^n{1}_{{ê}_i\le z},\text{and}{ĝ}_{h_1,a_1}^E(z)dz=d{Ĝ}_{h_1,a_1}(z)$$ (11)

leads to the nonparametric location-scale estimator of ∫|z|g_{h1, a1}(z)dz, $\int |z|{ĝ}_{h_1,a_1}^E(z)dz=n^{-1}{\sum }_{i=1}^n|\mathrm{m̂}(h_1,a_1)+\mathrm{\sigma ̂}(h_1,a_1){ê}_i|$. We show in Section 2·5 that the nonparametric location-scale estimator is consistent and asymptotically normal.

Data for estimating optimal sequential decision rules are typically expensive to collect, so samples are seldom large and parametric mean and variance function models are more useful than nonparametric models. Thus we assume that m(h₁, a₁) = m(h₁, a₁; θ) and σ(h₁, a₁) = σ(h₁, a₁; γ) for some θ ∈ ℝ^p_m and γ ∈ ℝ^p_s. Similarly, we assume that L(h₁, a₁) = L(h₁, a₁; α), α ∈ ℝ^p_L. For the results in Sections 3 we completed specification of the IQ-learning algorithm in Section 2·3 as follows. Steps IQ2a and IQ2b are amended to:

• IQ2aSet L̂(h₁, a₁) = L(h₁, a₁; α̂), where $\mathrm{\alpha ̂}={\text{arg min}}_{\alpha }{\sum }_{i=1}^n{\{\mathrm{\mu ̂}(H_{2,i})-L(H_{1,i},A_{1,i};\alpha )\}}^2$
• IQ2biSet m̂(h₁, a₁) = m(h₁, a₁;θ̂), where $\mathrm{\theta ̂}={\text{arg min}}_{\theta }{\sum }_{i=1}^n{\{\mathrm{\Delta ̂}(H_{2,i})-m(H_{1,i},A_{1,i};\theta )\}}^2$
• IQ2bii
  Set σ̂(h₁, a₁) = σ(h₁, a₁; γ̂) where

  $$\mathrm{\gamma ̂}={\text{arg min}}_{\gamma }{\sum }_{i=1}^n{\{\text{log}|\mathrm{\Delta ̂}(H_{2,i})-m(H_{1,i},A_{1,i};\mathrm{\theta ̂})|-\text{log}\sigma (H_{1,i},A_{1,i};\gamma )\}}^2.$$
• IQ2biii

  Set ĝ_{h1, a1} to either ${ĝ}_{h_1,a_1}^N$ in (9) or to ${ĝ}_{h_1,a_1}^E$ in (11).

We have used a simple model for the conditional variance in step IQ2bii; for a discussion of other conditional variance estimators and their asymptotic properties see Carroll and Ruppert (1988).

2·5. Asymptotic Theory

Asymptotic distribution theory for ${\mathrm{Q̂}}_2^{\mathrm{I}\mathrm{Q}}(h_1,a_1)$ is covered by standard results for linear regression, so we address only the asymptotic distribution of ${\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q}}(h_1,a_1)$ for the particular parametric estimators defined in the IQ-learning algorithm. Define the population residuals

$$ℇ(H_2,H_1,A_1;\theta ,\gamma ,{\beta }_2)=\{\mathrm{\Delta }(H_2;{\beta }_2)-m(H_1,A_1;\theta )\}/\sigma (H_1,A_1;\gamma )$$

and the population parameters:

$${\beta }_2^{*}=\text{arg}\underset{{\beta }_2}{\text{min}}E{\{Y-Q_2(H_2,A_1;{\beta }_2)\}}^2,{\theta }^{*}=\text{arg}\underset{\theta }{\text{min}}E{\{\mathrm{\Delta }(H_2;{\beta }_2^{*})-m(H_1,A_1;\theta )\}}^2,{\gamma }^{*}=\text{arg}\underset{\gamma }{\text{min}}E{\{\text{log}|\mathrm{\Delta }(H_2;{\beta }_2^{*})-m(H_1,A_1;{\theta }^{*})|-\text{log}\sigma (H_1,A_1;\gamma )\}}^2,{\alpha }^{*}=\text{arg}\underset{\alpha }{\text{min}}E{\{\mu (H_2;{\beta }_2^{*})-L(H_1,A_1;\alpha )\}}^2.$$

Let θ̂, γ̂, β̂₂, and α̂ denote n^1/2-consistent estimators of their population analogs θ*, γ*, ${\beta }_2^{*}$, and α*. For x ∈ ℝ^p, let ℬ_d(x) denote a ball of radius d centered at x, and let E_n denote the empirical expectation operator so that $E_{n}f=n^{-1}{\sum }_{i=1}^{n}f(H_{1,i},A_{1,i},H_{2,i},A_{2,i},Y_i)$. The asymptotic results are stated in terms of the seven centered statistics:

$${\mathrm{\Delta }}_L=L(h_1,a_1;\mathrm{\alpha ̂})-L(h_1,a_1;{\alpha }^{*}),{\mathrm{\Delta }}_m=m(h_1,a_1;\mathrm{\theta ̂})-m(h_1,a_1;{\theta }^{*}),{\mathrm{\Delta }}_{\beta }={\mathrm{\beta ̂}}_2-{\beta }_2^{*},{\mathrm{\Delta }}_{\sigma }=\sigma (h_1,a_1;\mathrm{\gamma ̂})-\sigma (h_1,a_1;{\gamma }^{*}),{\mathrm{\Delta }}_{\theta }=\mathrm{\theta ̂}-{\theta }^{*},{\mathrm{\Delta }}_{\gamma }=\mathrm{\gamma ̂}-{\gamma }^{*},{\mathrm{\Delta }}_{ℇ}=E_n\{|m(h_1,a_1;{\theta }^{*})+\sigma (h_1,a_1;{\gamma }^{*})ℇ(H_2,H_1,A_1;{\theta }^{*},{\gamma }^{*},{\beta }_2^{*})|\}-E\{|m(h_1,a_1;{\theta }^{*})+\sigma (h_1,a_1;{\gamma }^{*})ℇ(H_2,H_1,A_1;{\theta }^{*},{\gamma }^{*},{\beta }_2^{*})|\}$$

The following assumptions are used to establish the limit theory for IQ-learning:

• (A1N)n^1/2 (Δ_L, Δ_m, Δ_σ) is asymptotically N {0,Σ_N(h₁, a₁)};
• (A1E)n^1/2 (Δ_L, Δ_θ, Δ_γ, Δ_β, Δ_ε) is asymptotically N {0, Σ_E(h₁, a₁)};
• (A2)
  let k₁, k₂, and k₃ denote fixed positive constants, the class of functions

  $$\begin{gathered} ℱ=\{f(H_2,H_1,A_1;\theta ,\gamma ,{\beta }_2)=|m(h_1,a_1;\theta )+\sigma (h_1,a_1;\gamma )ℇ(H_2,H_1,A_1;\theta ,\gamma ,{\beta }_2)|: \\ \theta \in {ℬ}_{k_1}({\theta }^{*}),\gamma \in {ℬ}_{k_2}({\gamma }^{*}),{\beta }_2\in {ℬ}_{k_3}({\beta }_2^{*})\}, \end{gathered}$$

  is a P-measurable Donsker class with a square-integrable envelope. In addition, J(θ, γ, β₂) = Ef(H₂, H₁, A₁; θ, γ, β₂) is continuously differentiable with a bounded derivative in a neighborhood of $({\theta }^{*},{\gamma }^{*},{\beta }_2^{*})$;
• (A3)the random variable $ℰ(H_2,H_1,A_1;{\theta }^{*},{\gamma }^{*},{\beta }_2^{*})$ has a continuously differentiable density κ with derivative κ′(z) satisfying |∫z²κ′(z)dz| < ∞.

These assumptions are relatively mild with (A1N), (A1E), and the first part of (A2) verifiable using standard techniques, e.g., the multivariate central limit and Donsker preservation theorems (see Kosorok, 2008, and the Supplementary Material). Assumption (A3) is generally more difficult to verify, but its validity can be roughly assessed using the observed residuals ê₁,…, ê_n. We have not sought the most general assumptions, but rather a set of simple assumptions that illustrate what is needed for the IQ-learning estimator to be well-behaved.

The first result states the asymptotic normality of the normal and nonparametric location-scale IQ-learning estimators of the first-stage Q-function. Let ${\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q},N}(h_1,a_1)$ and ${\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q},E}(h_1,a_1)$ denote the normal and nonparametric location-scale estimators, respectively, so that

$${\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q},N}(h_1,a_1)=L(h_1,a_1;\mathrm{\alpha ̂})+\int |z|{ĝ}_{h_1,a_1}^N(z)dz,{\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q},E}(h_1,a_1)=L(h_1,a_1;\mathrm{\alpha ̂})+\int |z|{ĝ}_{h_1,a_1}^E(z)dz.$$

Define I(υ, t, s) = υ + s⁻¹∫|z|ϕ {(z − t)/s}dz. Let I*(h₁, a₁) denote I{L(h₁, a₁; α*), m(h₁, a₁; θ*), σ(h₁, a₁; γ*)}. The following is proved in the Supplementary Material.

Theorem 1 (Asymptotic normality.). Let h₁ and a₁ be fixed.

1. Assume (A1N). Then

   $$n^{1/2}[{\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q},N}(h_1,a_1)-L(h_1,a_1;{\alpha }^{*})-\frac{1}{\sigma (h_1,a_1;{\gamma }^{*})}\int |z|\varphi \{\frac{z-m(h_1,a_1;{\theta }^{*})}{\sigma (h_1,a_1;{\gamma }^{*})}\left\}dz\right]$$

   converges in distribution to N{0, ∇I* (h₁, a₁)^⊤Σ_N (h₁, a₁)∇I* (h₁, a₁)}.
2. Assume (A1E), (A2), and (A3). Then

   $$n^{1/2}[{\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q},E}(h_1,a_1)-L(h_1,a_1;{\alpha }^{*})-\frac{1}{\sigma (h_1,a_1;{\gamma }^{*})}\int |z|\kappa \{\frac{z-m(h_1,a_1;{\theta }^{*})}{\sigma (h_1,a_1;{\gamma }^{*})}\left\}dz\right]$$

   converges in distribution to $N[0,\{1,\nabla J{({\theta }^{*},{\gamma }^{*},{\beta }_2^{*})}^{\top },1\}{\mathrm{\sum }}_E(h_1,a_1){\{1,\nabla J{({\theta }^{*},{\gamma }^{*},{\beta }_2^{*})}^{\top },1\}}^{\top }]$.

Theorem 1 shows that both location-scale estimators ${\mathrm{Q̂}}_1^N(h_1,a_1)$ and ${\mathrm{Q̂}}_1^E(h_1,a_1)$ are asymptotically normal under the stated conditions, which do not require correct specification of the IQ-learning models. Under (C1) and (C2) below, the IQ-learning models are correctly specified, and consistency and asymptotic normality of the IQ-learning estimators follow.

• (C1)Let Z denote a standard normal random variable, then $\mathrm{\Delta }(H_2;{\beta }_2^{*})=m(H_1,A_1;{\theta }^{*})+\sigma (H_1,A_1;{\gamma }^{*})$.
• (C2)Let W be a random variable with density κ(·), then $\mathrm{\Delta }(H_2;{\beta }_2^{*})=m(H_1,A_1;{\theta }^{*})+\sigma (H_1,A_1;{\gamma }^{*})W$.

The following results are direct consequences of Theorem 1 and we omit their proofs.

Corollary 1. Assume $Q_2(h_2,a_2)=Q_2(h_2,a_2;{\beta }_2^{*})$ and $E\{\mu (H_2;{\beta }_2^{*})|H_1=h_1,A_1=a_1\}=L(h_1,a_1;{\alpha }^{*})$. Then

1. if (C1), $Q_1(h_1,a_1)=L(h_1,a_1;{\alpha }^{*})+\frac{1}{\sigma (h_1,a_1;{\gamma }^{*})}\int |z|\varphi \left\{\frac{z-m(h_1,a_1;{\theta }^{*})}{\sigma (h_1,a_1;{\gamma }^{*})}\right\}dz$;

2. if (C2), $Q_1(h_1,a_1)=L(h_1,a_1;{\alpha }^{*})+\frac{1}{\sigma (h_1,a_1;{\gamma }^{*})}\int |z|\kappa \left\{\frac{z-m(h_1,a_1;{\theta }^{*})}{\sigma (h_1,a_1;{\gamma }^{*})}\right\}dz$.

Theorem 2. Assume (A1N) and the conditions of Theorem 1. Let h₁ and a₁ be fixed.

1. If (C1) then $n^{1/2}\{{\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q},N}(h_1,a_1)-Q_1(h_1,a_1)\}$ converges in distribution to N{0, ∇I*(h₁, a₁)^⊤Σ_N (h₁, a₁)∇I*(h₁, a₁)}.

2. If (A1E), (A2), (A3) and (C2) then $n^{1/2}\{{\mathrm{Q̂}}_1^{\mathrm{I}\mathrm{Q},E}(h_1,a_1)-Q_1(h_1,a_1)\}$ converges in distribution to $N[0,\{1,\nabla J{({\theta }^{*},{\gamma }^{*},{\beta }_2^{*})}^{\top },1\}{\mathrm{\Sigma }}_E(h_1,a_1){\{1,\nabla J{({\theta }^{*},{\gamma }^{*},{\beta }_2^{*})}^{\top },1\}}^{\top }]$.

Remark 1. Theorem 2 can be used to construct asymptotically valid confidence intervals for the first-stage Q-function for fixed patient history h₁ and first-stage treatment a₁. This is a notoriously difficult task with Q-learning (Laber et al., 2014). In practice, due to the complexity of the variance terms, the bootstrap may be preferred. Remarks about how to extend the above results to obtain bootstrap consistency are made in the Supplementary Material.

Remark 2. As noted in the introduction, IQ-learning does not alleviate the inherent nonregularity present in sequential decision making problems; see Robins (2004), Laber et al. (2014), and Chakraborty et al. (2010). However, IQ-learning is consistent for a nonregular scenario of interest, the so-called global null in which there is no treatment effect for any patients at the second stage, i.e., $\mathrm{\Delta }(H_2;{\beta }_2^{*})=0$ almost surely. To see this, note that assuming (A1N), (C1) holds with m(H₁, A₁; θ*) = 0 with σ(H₁, A₁; γ*) → 0 almost surely. Part 1 of Corollary 1 depends only on part 1 of Theorem 1 and part 1 of Theorem 2. For the more complex case in which $0<\mathrm{p}\mathrm{r}\{\mathrm{\Delta }(H_2;{\beta }_2^{*})=0\}<1$ we conjecture that using a mixture of normals to estimate g_h1,a1(·) may lead to improved small–sample performance.

3. Monte Carlo Results

We compare the small-sample performance of IQ- and Q-learning in using average value of the learned treatment regimes, integrated mean squared error of the first-stage Q-function, and coverage and width of 95% nonparametric bootstrap confidence intervals for the first-stage Q-function. A key advantage of IQ-learning is its compatibility with common model building steps, which we illustrate in the Supplementary Material with a study of the power to detect nonlinear effects in the first-stage Q-function using IQ-learning and Q-learning. Software implementing the IQ-learning estimators is available as part of the iqLearn package on the comprehensive R network (cran.us.r-project.org/). The Monte Carlo results show that for the class of generative models we consider, IQ-learning generally performs better than Q-learning in terms of value, integrated mean squared error, coverage, and width of the confidence intervals. Simulations in the Supplementary Material show that IQ-learning also has higher power to detect nonlinear effects.

Simulations in this section use data from the following class of generative models:

$$X_1~{\text{Normal}}_p\{0.1,{\mathrm{\Omega }}_{A{R}_1}(0.5)\},A_t~\text{Uniform}\{-1,1\},t=1,2,X_2=(1.5-0.5A_1)X_1+{\zeta }_{A_1}\xi ,Y=H_{2,0}^{\top }{\beta }_{2,0}+A_2{H}_{2,1}^{\top }{\beta }_{2,1}+\varphi ,$$

where {Ω_AR1 (0.5)}_i,j = (0.5)^|i−j|, $H_{2,0}=H_{2,1}={(1,X_2^{\top },A_1,A_1{X}_2^{\top })}^{\top }$, and ζ_A1 = (1.5 + 0.5A₁)^1/2. Thus, the class is indexed by the dimension p, the distributions of ξ and ϕ, and the coefficient vectors β_2,0 and β_2,1. Here we fix p = 4; results for p = 8 are similar and are provided in the Supplementary Material. We consider ξ ~ Normal_p (0, I_p). We fix the main effect parameter β_2,0 and vary the second-stage treatment effect size by scaling β_2,1 as follows: β_2,0 = 1_2p+2/‖1_2p+2‖, ${\beta }_{2,1}={C\{-0.25(1_{p+1}^{\top }),1_{p+1}^{\top }\}}^{\top }/\Vert \{-0.25(1_{p+1}^{\top }),1_{p+1}^{\top }\}\Vert$, where C ranges over a grid from 0 to 2, and 1_d denotes a d-dimensional vector of 1s. In addition, we fix the theoretical R² of the second-stage regression model at 0.6 by specifying $\varphi ~\text{Normal}\{0,{\sigma }_{\varphi }^2(C)\}$ and solving for the variance ${\sigma }_{\varphi }^2(C)$ that yields the desired R². Additional simulations, provided in the Supplementary Material, show results for R² = 0.4, 0.8 and non-normal error distributions for ξ;

We consider linear working models for the mean and log variance functions

$$\begin{array}{cccc}{Q}_2(h_2,a_2;{\beta }_2)\hfill & =h_2^{\top }{\beta }_{2,0}+a_2{h}_2^{\top }{\beta }_{2,1},\hfill & Q_1(h_1,a_1;{\beta }_1)\hfill & =h_1^{\top }{\beta }_{1,0}+a_1{h}_1^{\top }{\beta }_{1,1},\hfill \\ L(h_1,a_1;\alpha )\hfill & =h_1^{\top }{\alpha }_0+a_1{h}_1^{\top }{\alpha }_1,\hfill & m(h_1,a_1;\theta )\hfill & =h_1^{\top }{\theta }_0+a_1{h}_1^{\top }{\theta }_1,\hfill \\ \text{log}\{\sigma (h_1,a_1;\gamma )\}\hfill & =h_1^{\top }{\gamma }_0+a_1{h}_1^{\top }{\gamma }_1,\hfill & \hfill \end{array}$$

where now $H_1={(1,X_1^{\top })}^{\top }$. In addition to Q-learning with linear working models, we include results using support vector regression using a Gaussian kernel (Zhao et al., 2011) to estimate both Q-functions. We compare the two versions of Q-learning with two IQ-learning estimators that differ in the estimation of g_h1,a1(·) and the model for σ(h₁, a₁; γ): a normal estimator ${ĝ}_{h_1,a_1}^N(·)$ of the residual distribution and a restricted variance model, log{σ(h₁, a₁; γ)} = γ₀ + a₁γ₁, that depends only on treatment; and a nonparametric estimator ${ĝ}_{h_1,a_1}^E(·)$ of the residual distribution with a log-linear variance model that depends on h₁ and a₁. When ξ ~ Normal_p(0, I_p), both these estimators are correctly specified. Q-learning is always correctly specified at the second stage but only correctly specified at the first stage when C = 0 and hence β_2,1 = 0.

Results are based on a training set of size n = 250 and M = 2,000 Monte Carlo data sets for each generative model. Additional results for n = 500 are provided in the Supplementary Material. For the nonparametric IQ-learning estimator, which is always correctly specified, the true Q-functions and subsequent optimal regime are estimated using a test set of 10,000 observations. Recall that the value, E^π (Y), of an arbitrary policy π is the expected outcome if all patients are assigned treatment according to π, that is, E^π (Y) = E(E [E {Y | H₂, a₂} | H₁, a₁]) evaluated at a₂ = π₂(H₂) and a₁ = π₁(H₁). For a given training set of size n and an algorithm that produces an estimated optimal policy, say π̂, we define the average value as E {E^π̂ (Y)}, where the outer expectation is taken over all training sets of size n. We estimate the average value of the IQ- and Q-learning estimators using a test set of size 10,000 to estimate the inner expectation and 2,000 Monte Carlo replications to estimate the outer expectation. We compare average values of the learned IQ and Q regimes to the value of the true optimal regime and present the proportion of optimal value obtained. For an estimator Q̂₁(h₁, a₁) of the first-stage Q-function, Q₁(h₁, a₁), define the integrated mean squared error as E [{Q̂₁(H₁, A₁) − Q₁(H₁, A₁)}²], where the expectation is taken over the joint distribution of (H₁, A₁) as well as the training data.

Confidence intervals for Q₁(h₁, a₁) based on IQ-learning and Q-learning estimators are formed by bootstrapping the respective estimators and taking percentiles. For example, if l̂ and û denote the 100 × η/2 and 100 × (1 − η/2) percentiles of the bootstrap distribution of Q̂^IQ(h₁, a₁) based on 1,000 bootstrap resamples, then the 100 × (1 − η)% confidence interval is given by (2Q̂^IQ(h₁, a₁) − û, 2Q̂^IQ(h₁, a₁) − l̂). Bootstrap intervals of this form are sometimes referred to as hybrid bootstrap confidence intervals (Efron and Tibshirani, 1993). Coverage and width of the foregoing confidence intervals are estimated using 2,000 Monte Carlo replications with a new instance (h₁, a₁) of (H₁, A₁) drawn for each replication. Figure 2 displays the results from this simulation, where ξ ~ Normal_p(0, I_p).

Fig. 2.

Performance of the normal IQ-learning estimator, nonparametric IQ-learning estimator, support vector regression Q-learning, and linear Q-learning given by gray lines with squares, gray lines with circles, light gray lines, and black lines, respectively. Left to Right: Average proportion of optimal value attained; integrated mean squared error of Q₁ estimates; coverage of 95% confidence intervals for Q₁; width of 95% confidence intervals for Q₁.

Figure 2 indicates that the IQ-learning estimators perform better than both Q-learning estimators. Although some gains are achieved using the more flexible support vector regression version of Q-learning, the far left panel indicates that IQ-learning attains higher average value than the Q-learning algorithms across most values of C. The second panel from the left shows that the IQ-learning reduces integrated mean squared error the most, with greater reduction as the second-stage effects increase. IQ-learning also demonstrates a large improvement over linear Q-learning in terms of the coverage of 95% confidence intervals for Q̂₁ (h₁, a₁), as seen in the third panel. The poor coverage of linear Q-learning is attributed to bias, whereas IQ-learning is consistent because the regularity conditions given in Section 2.4 for Theorem 2 hold for the generative models in this section. Thus, IQ-learning estimators come close to achieving the nominal level. The average widths of the confidence intervals are similar for linear Q-learning and IQ-learning, as illustrated by the far right panel. Support vector regression Q-learning greatly improves coverage compared to linear Q-learning, but at the expense of wider intervals. Results from the simulation where the elements of ξ are generated independently from a t₅ distribution are included in the Supplementary Material and appear similar to those in Fig. 2, suggesting the normal IQ-learning estimator is robust to slight misspecification of the residual distribution.

Next, we consider the model

$$\begin{array}{ccc}{X}_1~\text{Normal}(-2,1),\hfill & \xi ~\text{Normal}(0,1),\hfill & A_t~\text{Uniform}\{-1,1\},t=1,2,\hfill \\ X_2=X_1+\xi ,\hfill & \varphi ~\text{Normal}(0,1),\hfill & Y=H_{2,0}^{\top }{\beta }_{2,0}+A_2{H}_{2,1}^{\top }{\beta }_{2,1}+\varphi ,\hfill \end{array}$$

where H_2,0 = H_2,1 = (1, X₁, A₁, X₁A₁, X₂)^⊤, β_2,0 = (3, −1, 0.1, −0.1, −0.1)^⊤, and β_2,1 = C(−6, −2, 5, 3, −0.2)^⊤. We use this example to illustrate a scenario where IQ-learning achieves a large gain in value over Q-learning with linear models. Since the predictor X₁ is univariate, we can visualize which patients are treated differently by IQ-learning compared to Q-learning. The left plot in Figure 3 was obtained by deriving the true first stage Q-function, for which IQ-learning is consistent in this case, and comparing the true first-stage rule to the rule recommended by Q-learning with linear models. For each combination of (X₁, C), the plot shows whether or not Q-learning makes the correct treatment decision. With this generative model, Q-learning assigns the wrong treatment to approximately half the population for a wide range of effect sizes because the true first-stage Q-function is nonlinear in X₁. In contrast, with a sufficiently large sample size, IQ-learning treats all patients according to the optimal rule. A remark explaining the observed pattern is included in Section 6 of the Supplementary Material. Consequently, IQ-learning achieves higher average value than Q-learning, as displayed in the right plot of Fig. 3. Results are based on n = 250 training set samples and M = 1,000 Monte Carlo data sets. In this scenario, both IQ-learning and support vector regression Q-learning reach gains in optimal value attained of approximately 15% as the second-stage effect size grows.

Fig. 3.

A scenario where IQ-learning achieves a large gain in value over Q-learning. The constant C determines the second-stage treatment effect size, from no treatment effects (C = 0) to large effects (C = 2). In the left panel, (X₁, C) pairs where linear Q-learning agrees and disagrees with the true first-stage rule are shown in dark and light gray, respectively, where X₁ is a normally distributed first-stage covariate. On the right, average proportion of optimal value attained by the normal IQ-learning estimator, nonparametric IQ-learning estimator, support vector regression Q-learning, and linear Q-learning regimes shown by gray lines with squares, gray lines with circles, light gray lines, and black lines, respectively.

4. Application to STAR*D

Sequenced Treatment Alternatives to Relieve Depression (STAR*D; Fava et al., 2003; Rush et al., 2004) is a sequentially randomized study of major depressive disorder. A key feature of this trial is that patients experiencing early remission of symptoms were exempt from future randomization, complicating the analysis. We use a subset of the STAR*D data to illustrate how IQ-learning can be used to estimate an optimal dynamic treatment regime in the presence of responder-status dependent designs. There were four stages in the trial, but each patient received Citalopram in the first stage, and thus, there was no randomization. Since our aim is to demonstrate how to learn a regime using the IQ-learning machinery, we opt to perform a complete-case analysis and consider only the first two of three randomized stages. We refer to the second and third stages as stages one and two, respectively. At each stage, treatments can be categorized as a Selective Serotonin Reuptake Inhibitor or not; this is the binary treatment variable in our analysis. The first-line treatment Citalopram given in the non-randomized stage is in the class of Selective Serotonin Reuptake Inhibitors.

We use a measure of efficacy, the Quick Inventory of Depression Symptomatology score, as the outcome (Rush et al., 2004); side-effects or other competing outcomes could be accommodated using set-valued treatment regimes (Lizotte et al., 2012; Laber et al., 2013) or composite outcomes (Wang et al., 2012). The depression score ranges from 0 to 27 with higher values corresponding to more severe negative symptoms. To be consistent with our development, we recode these scores by subtracting them from 27; thus, higher values correspond to better clinical outcomes. The depression score was recorded at multiple time points throughout each stage, intermediate depression scores used as predictors in our analysis are also recoded (Rush et al., 2004, Schulte et al., 2012). At each stage, patients experiencing remission left the study. Remission was defined as a depression score of 5 or less, or greater than 21 after recoding. Henceforth, we refer to patients who achieved remission and left after stage one as responders and second-stage participants as non-responders. The data we use here consist of n = 795 patient trajectories with complete information, not including 481 patients who dropped out for reasons other than remission. The variables composing each trajectory are given in Table 1 of the Supplementary Material.

Of the 795 total patients, 329 were non-responders in the first stage and continued on to the second stage. We define our primary outcome as Y = RY₁ + (1 − R)(Y₁ + Y₂)/2. That is, Y is taken to be Y₁ for patients who left the study after stage one, and Y is taken to be the average of the depression scores measured at the end of the first and second stages for non-responders. Our responder-status version of IQ-learning is based on the Q-learning implementation described in Schulte et al. (2012). The first-stage history vector contains all information available prior to the first-stage treatment randomization. Thus, H₁ = (X_1,1, X_1,2)^⊤. The second-stage history is H₂ = (X_1,1, X_1,2, A₁, Y₁, R, X_2,1, X_2,2)^⊤, which contains all information observed before the second-stage treatment assignment. The second-stage Q-function is Q₂(H₂, A₂) = E(Y | H₂ = h₂, A₂ = a₂) = RY₁ + (1 − R) {Y₁ + E(Y₂ | H₂ = h₂, A₂ = a₂)} /2, where we have used the fact that R and Y₁ are contained in H₂. Thus the first step in the IQ-learning algorithm, and in Q-learning, is to specify and fit a model for E(Y₂ | H₂ = h₂, A₂ = a₂). Defining the second-stage summary vectors as H_2,0 = H_2,1 = (1, X_2,1, X_2,2)^⊤, we consider a working model of the form $E(Y_2|H_2=h_2,A_2=a_2)=h_{2,0}^{\top }{\beta }_{2,0}+a_2{h}_{2,1}^{\top }{\beta }_{2,1}$, that we fit via least squares. Standard regression diagnostics based on the residuals to not indicate any major departures from the usual linear modeling assumptions.

In our subset of data, all non-responders who did not receive a Selective Serotonin Reuptake Inhibitor in stage one did not receive one in stage two. Table 2 of the Supplementary Material provides the number of patients assigned to each treatment strategy. We define Ỹ = arg max_a2 Q₂(H₂, a₂)R + {Q₂(H₂, −1)(1 − A₁) + arg max_a2 Q₂(H₂, a₂)(1 + A₁)} (1 − R)/2. Thus, Ỹ is Q₂(H₂, −1) for non-responders who received A₁ = −1 and arg max_a2 Q₂(H₂, a₂) otherwise. With this working model, after substituting in Q₂(H₂, a₂) and simplifying,

$$Q_1(H_1,A_1)=E(R{Y}_1|H_1,A_1)+\frac{1}{2}E\{(1-R)Y_1|H_1,A_1\}+\frac{1}{2}E\{(1-R)H_{2,0}^{\top }{\beta }_{2,0}|H_1,A_1\}-\frac{1-A_1}{4}E\{(1-R)H_{2,1}^{\top }{\beta }_{2,1}|H_1,A_1\}+\frac{1+A_1}{4}E\{(1-R)|H_{2,1}^{\top }{\beta }_{2,1}||H_1,A_1\}.$$ (12)

The Q-learning algorithm models E(Ỹ | H₁, A₁) directly. The left panel in Figure 4 shows a scatterplot of the pseudo-response Ỹ against baseline depression score by first-stage treatment. Cubic smoothing spline fits to the data are indicated by solid gray and dashed black lines for A₁ = 1 and A₁ = −1, respectively. These fits appear approximately linear; however, the variance appears non-constant across baseline depression score, and there is clear separation between the responder and non-responder groups.

Fig. 4.

First-stage summaries of the STAR*D data. Gray circles and black crosses represent treatments A₁ = 1 and A₁ = −1, respectively. The left panel shows a scatterplot of Ỹ against baseline depression score by first-stage treatment. The middle panel contains a scatterplot of $H_{2,0}^{\top }{\mathrm{\beta ̂}}_{2,0}$ against baseline depression score by first-stage treatment for non-responders. The right panel contains a scatterplot of $H_{2,1}^{\top }{\mathrm{\beta ̂}}_{2,1}$ against baseline depression score by first-stage treatment for non-responders. Cubic smoothing spline fits to the data for A₁ = 1 and A₁ = −1 are represented by solid gray and dashed black lines, respectively.

We can write the first three expectation terms in (12) as:

$$\begin{array}{cc}\hfill E(R{Y}_1|H_1,A_1)& =E(Y_1|H_1,A_1,R=1)\text{pr}(R=1|H_1,A_1),\hfill \\ \hfill E\{(1-R)Y_1|H_1,A_1\}& =E(Y_1|H_1,A_1,R=0)\text{pr}(R=0|H_1,A_1),\hfill \\ \hfill E\{(1-R)H_{2,0}^{\top }{\beta }_{2,0}|H_1,A_1\}& =E(H_{2,0}^{\top }{\beta }_{2,0}|H_1,A_1,R=0)\text{pr}(R=0|H_1,A_1).\hfill \end{array}$$

We estimate the right-hand conditional expectations by fitting three separate linear regressions, and we use logistic regression to estimate pr(R = r | H₁, A₁, r = 0, 1. In particular, we specify linear models of the form $E(Y_1|H_1=h_1,A_1=a_1,R=r)=h_{1,0}^{\top }{\lambda }_{r,0}+a_1{h}_{1,1}^{\top }{\lambda }_{r,1}$ for E(Y₁ | H₁, A₁, R = r), where H_1,0 = H_1,1 = (1, X_1,1, X_1,2)^⊤. We posit the model $E(H_{2,0}^{\top }{\beta }_{2,0}|H_1=h_1,A_1=a_1,R=0)=h_{1,0}^{\top }{\alpha }_0+a_1{h}_{1,1}^{\top }{\alpha }_1$ for the main-effect term, which we fit with least squares using only the non-responder data. The middle and right plots in Fig. 4 display scatterplots of the non-responder realizations of the main-effect term and contrast function, respectively, against baseline depression score by first-stage treatment. Cubic smoothing spline fits to the data appear mostly linear. For the logistic regression, we fit the model $\text{logit}\{\text{pr}(R=1|H_1=h_1,A_1=a_1)\}=h_{1,0}^{\top }{\delta }_0+a_1{h}_{1,1}^{\top }{\delta }_1$.

Finally, we must obtain estimates of $E\{(1-R)|H_{2,1}^{\top }{\beta }_{2,1}||H_1,A_1\}$ and $E\{(1-R)H_{2,1}^{\top }{\beta }_{2,1}|H_1,A_1\}$ in equation (12). Notice that

$$E\{(1-R)|H_{2,1}^{\top }{\beta }_{2,1}||H_1,A_1\}=\text{pr}(R=0|H_1,A_1)\int |z|g_{H_1,A_1,R=0}(z)dz.$$

We can use the IQ-learning machinery to obtain an estimate of g_{H1, A1, R=0}(·). The logistic regression previously described provides an estimate of pr(R = 0 | H₁, A₁). To estimate g_{H1, A1, R=0}(·), we first specify a model for the mean of the contrast function for non-responders. We posit the model

$$E\{\mathrm{\Delta ̂}(H_2;{\mathrm{\beta ̂}}_2)|H_1,A_1\}=H_{1,0}^{\top }{\beta }_0+A_1{H}_{1,1}^{\top }{\beta }_1,$$ (13)

where H_1,0 = (1, X_1,1, X_1,2)^⊤ and H_1,1 = (1, X_1,1, X_1,2)^⊤. This model is also used along with the logistic regression model to estimate $E\{(1-R)H_{2,1}^{\top }{\beta }_{2,1}|H_1,A_1\}=E(H_{2,1}^{\top }{\beta }_{2,1}|H_1,A_1,R=0)\times \text{pr}(R=0|H_1,A_1)$, the fourth expectation term in (12). We fit model (13) using least squares. Exploratory analysis suggests that a constant variance assumption is reasonable, so we standardize the residuals of the mean fit using the sample standard deviation. A normal quantile-quantile plot of the standardized residuals suggests heavier tails than would be expected from a normal distribution. Thus, we opt to use the nonparametric density estimator described in Section 2.3.

Assembling the foregoing estimates of the four terms in equation (12) yields

$${\mathrm{Q̂}}_1^{IQ}(h_1,a_1;{\mathrm{\theta ̂}}_1)=(h_{1,0}^{\top }{\mathrm{\lambda ̂}}_{1,0}+a_1{h}_{1,1}^{\top }{\mathrm{\lambda ̂}}_{1,1})\text{expit}(h_{1,0}^{\top }{\mathrm{\delta ̂}}_0+a_1{h}_{1,1}^{\top }{\mathrm{\delta ̂}}_1)+\{1-\text{expit}(h_{1,0}^{\top }{\mathrm{\delta ̂}}_0+a_1{h}_{1,1}^{\top }{\mathrm{\delta ̂}}_1)\}\{\frac{1}{2}(h_{1,0}^{\top }{\mathrm{\lambda ̂}}_{0,0}+a_1{h}_{1,1}^{\top }{\mathrm{\lambda ̂}}_{0,1})+\frac{1}{2}(h_{1,0}^{\top }{\mathrm{\alpha ̂}}_0+a_1{h}_{1,1}^{\top }{\mathrm{\alpha ̂}}_1)-\frac{1}{4}(h_{1,0}^{\top }{\mathrm{\beta ̂}}_0+a_1{h}_{1,1}^{\top }{\mathrm{\beta ̂}}_1)+\frac{1}{4}\int |z|{ĝ}_{h_1,a_1,R=0}(z)dz\},$$

where ĝ_{h1, a1, R=0}(·) is the nonparametric density estimator described in Section 2.3 and ${\mathrm{\theta ̂}}_1={({\mathrm{\lambda ̂}}_{0,0}^{\top },{\mathrm{\lambda ̂}}_{0,1}^{\top },{\mathrm{\lambda ̂}}_{1,0}^{\top },{\mathrm{\lambda ̂}}_{1,1}^{\top },{\mathrm{\delta ̂}}_0^{\top },{\mathrm{\delta ̂}}_1^{\top },{\mathrm{\alpha ̂}}_0^{\top },{\mathrm{\alpha ̂}}_1^{\top },{\mathrm{\beta ̂}}_0^{\top },{\mathrm{\beta ̂}}_1^{\top })}^{\top }$.

The first-stage rule estimated by Q-learning treats all training data patients with a Selective Serotonin Reuptake Inhibitor. Roughly twelve percent of the these patients are not recommended a Selective Serotonin Reuptake Inhibitor by the estimated first-stage IQ-learning rule. Broadly summarizing the IQ-learning rule, patients with very low recoded depression scores after the non-randomized stage should switch from Citalopram, a Selective Serotonin Reuptake Inhibitor, to another drug not in the class of Selective Serotonin Reuptake Inhibitors. This rule recommends a different strategy for patients who respond poorly to the initial Selective Serotonin Reuptake Inhibitor; otherwise Selective Serotonin Reuptake Inhibitor treatment strategies should continue.