PERFORMANCE GUARANTEES FOR INDIVIDUALIZED TREATMENT RULES

Abstract

Because many illnesses show heterogeneous response to treatment, there is increasing interest in individualizing treatment to patients [11]. An individualized treatment rule is a decision rule that recommends treatment according to patient characteristics. We consider the use of clinical trial data in the construction of an individualized treatment rule leading to highest mean response. This is a difficult computational problem because the objective function is the expectation of a weighted indicator function that is non-concave in the parameters. Furthermore there are frequently many pretreatment variables that may or may not be useful in constructing an optimal individualized treatment rule yet cost and interpretability considerations imply that only a few variables should be used by the individualized treatment rule. To address these challenges we consider estimation based on l₁ penalized least squares. This approach is justified via a finite sample upper bound on the difference between the mean response due to the estimated individualized treatment rule and the mean response due to the optimal individualized treatment rule.

1. Introduction

Many illnesses show heterogeneous response to treatment. For example, a study on schizophrenia [12] found that patients who take the same antipsychotic (olanzapine) may have very different responses. Some may have to discontinue the treatment due to serious adverse events and/or acutely worsened symptoms, while others may experience few if any adverse events and have improved clinical outcomes. Results of this type have motivated researchers to advocate the individualization of treatment to each patient [16, 24, 11]. One step in this direction is to estimate each patient’s risk level and then match treatment to risk category [5, 6]. However, this approach is best used to decide whether to treat; otherwise it assumes the knowledge of the best treatment for each risk category. Alternately, there is an abundance of literature focusing on predicting each patient’s prognosis under a particular treatment [10, 28]. Thus an obvious way to individualize treatment is to recommend the treatment achieving the best predicted prognosis for that patient. In general the goal is to use data to construct individualized treatment rules that, if implemented in future, will optimize the mean response.

Consider data from a single stage randomized trial involving several active treatments. A first natural procedure to construct the optimal individualized treatment rule is to maximize an empirical version of the mean response over a class of treatment rules (assuming larger responses are preferred). As will be seen, this maximization is computationally difficult because the mean response of a treatment rule is the expectation of a weighted indicator that is non-continuous and non-concave in the parameters. To address this challenge we make a substitution. That is, instead of directly maximizing the empirical mean response to estimate the treatment rule, we use a two-step procedure that first estimates a conditional mean and then from this estimated conditional mean derives the estimated treatment rule. As will be seen in Section 3, even if the optimal treatment rule is contained in the space of treatment rules considered by the substitute two-step procedure, the estimator derived from the two-step procedure may not be consistent. However if the conditional mean is modeled correctly, then the two-step procedure consistently estimates the optimal individualized treatment rule. This motivates consideration of rich conditional mean models with many unknown parameters. Furthermore there are frequently many pretreatment variables that may or may not be useful in constructing an optimal individualized treatment rule, yet cost and interpretability considerations imply that fewer rather than more variables should be used by the treatment rule. This consideration motivates the use of l₁ penalized least squares (l₁-PLS).

We propose to estimate an optimal individualized treatment rule using a two step procedure that first estimates the conditional mean response using l₁-PLS with a rich linear model and second, derives the estimated treatment rule from estimated conditional mean. For brevity, throughout, we call the two step procedure the l₁-PLS method. We derive several finite sample upper bounds on the difference between the mean response to the optimal treatment rule and the mean response to the estimated treatment rule. All of the upper bounds hold even if our linear model for the conditional mean response is incorrect and to our knowledge are, up to constants, the best available. We use the upper bounds in Section 3 to illuminate the potential mismatch between using least squares in the two-step procedure and the goal of maximizing mean response. The upper bounds in Section 4.1 involve a minimized sum of the approximation error and estimation error; both errors result from the estimation of the conditional mean response. We shall see that l₁-PLS estimates a linear model that minimizes this approximation plus estimation error sum among a set of suitably sparse linear models.

If the part of the model for the conditional mean involving the treatment effect is correct, then the upper bounds imply that, although a surrogate two-step procedure is used, the estimated treatment rule is consistent. The upper bounds provide a convergence rate as well. Furthermore in this setting the upper bounds can be used to inform how to choose the tuning parameter involved in the l₁-penalty to achieve the best rate of convergence. As a by-product, this paper also contributes to existing literature on l₁-PLS by providing a finite sample prediction error bound for the l₁-PLS estimator in the random design setting without assuming the model class contains or is close to the true model.

The paper is organized as follows. In Section 2, we formulate the decision making problem. In Section 3, for any given decision, e.g. individualized treatment rule, we relate the reduction in mean response to the excess prediction error. In Section 4, we estimate an optimal individualized treatment rule via l₁-PLS and provide a finite sample upper bound on the maximal reduction in optimal mean response achieved by the estimated rule. In Section 5, we consider a data dependent tuning parameter selection criterion. This method is evaluated using simulation studies and illustrated with data from the Nefazodone-CBASP trial [13]. Discussions and future work are presented in Section 6.

2. Individualized treatment rules

We use upper case letters to denote random variables and lower case letters to denote values of the random variables. Consider data from a randomized trial. On each subject we have the pretreatment variables X ∈ , treatment A taking values in a finite, discrete treatment space , and a real-valued response R (assuming large values are desirable). An individualized treatment rule (ITR) d is a deterministic decision rule from into the treatment space .

Denote the distribution of (X, A, R) by P. This is the distribution of the clinical trial data; in particular, denote the known randomization distribution of A given X by p(·|X). The likelihood of (X, A, R) under P is then f₀(x)p(a|x)f₁(r|x, a), where f₀ is the unknown density of X and f₁ is the unknown density of R conditional on (X, A). Denote the expectations with respect to the distribution P by an E. For any ITR d : → , let P^d denote the distribution of (X, A, R) in which d is used to assign treatments. Then the likelihood of (X, A, R) under P^d is f₀(x)1_a=d(x)f₁(r|x, a). Denote expectations with respect to the distribution P^d by an E^d. The Value of d is defined as V (d) = E^d(R). An optimal ITR, d₀, is a rule that has the maximal Value, i.e.

$$d_0\in arg\underset{d}{max}V(d),$$

where the argmax is over all possible decision rules. The Value of d₀, V(d₀), is the optimal Value.

Assume P[p(a|X) > 0] = 1 for all a ∈ (i.e. all treatments in are possible for all values of X a.s.). Then P^d is absolutely continuous with respect to P and a version of the Radon-Nikodym derivative is dP^d/dP = 1_a=d(x)/p(a|x). Thus the Value of d satisfies

$$V(d)=E^d(R)=\int {\mathit{RdP}}^d=\int R\frac{{dP}^d}{dP}dP=E\left[\frac{1_{A=d(X)}}{p(A\mid X)}R\right].$$ (2.1)

Our goal is to estimate d₀, i.e. the ITR that maximizes (2.1), using data from distribution P. When X is low dimensional and the best rule within a simple class of ITRs is desired, empirical versions of the Value can be used to construct estimators [21, 27]. However if the best rule within a larger class of ITRs is of interest, these approaches are no longer feasible.

Define Q₀(X,A) ≜ E(R|X,A) (Q₀(X,A) is sometimes called the “Quality” of treatment a at observation x). It follows from (2.1) that for any ITR d,

$$V(d)=E\left[\frac{1_{A=d(X)}}{p(A\mid X)}{Q}_0(X,A)\right]=E\left[\sum _{a\in \mathcal{A}}{1}_{d(X)=a}{Q}_0(X,a)\right]=E[Q_0(X,d(X))].$$

Thus V (d₀) = E[Q₀(X, d₀(X))] ≤ E[max_a∈Q₀(X, a)]. On the other hand, by the definition of d₀,

$$V(d_0)\ge V(d){|}_{d(X)\in arg{max}_{a\in \mathcal{A}}{Q}_0(X,a)}=E\left[\underset{a\in \mathcal{A}}{max}{Q}_0(X,a)\right].$$

Hence an optimal ITR satisfies d₀(X) ∈ arg max_a∈ Q₀(X, a) a.s.

3. Relating the reduction in Value to excess prediction error

The above argument indicates that the estimated ITR will be of high quality (i.e. have high Value) if we can estimate Q₀ accurately. In this section, we justify this by providing a quantitative relationship between the Value and the prediction error.

Because is a finite, discrete treatment space, given any ITR, d, there exists a square integrable function Q : × → ℝ for which d(X) ∈ arg max_a Q(X, a) a.s. Let L(Q) ≜ E[R − Q(X,A)]² denote the prediction error of Q (also called the mean quadratic loss). Suppose that Q₀ is square integrable and that the randomization probability satisfies p(a|x) ≥ S⁻¹ for an S > 0 and all (x, a) pairs. Murphy [23] showed that

$$V(d_0)-V(d)\le 2S^{1/2}{[L(Q)-L(Q_0)]}^{1/2}.$$ (3.1)

Intuitively, this upper bound means that if the excess prediction error of Q (i.e. E(R − Q)² − E(R − Q₀)²) is small, then the reduction in Value of the associated ITR d (i.e. V(d₀) − V(d)) is small. Furthermore the upper bound provides a rate of convergence for an estimated ITR. For example, suppose Q₀ is linear, that is Q₀ = Φ(X, A)θ₀ for a given vector-valued basis function Φ on × and an unknown parameter θ₀. And Suppose we use a correct linear model for Q₀ (here “linear” means linear in parameters), say the model = {Φ(X, A)θ : θ → ℝ^dim(Φ)} or a linear model containing with dimension of parameters fixed in n. If we estimate θ by least squares and denote the estimator by θ̂, then the prediction error of Q̂ = Φθ̂ converges to L(Q₀) at rate 1/n under mild regularity conditions. This together with inequality (3.1) implies that the Value obtained by the estimated ITR, d̂(X) ∈ arg max_a Q̂(X, a), will converge to the optimal Value at rate at least $1/\sqrt{n}$.

In the following theorem, we improve this upper bound in two aspects. First, we show that an upper bound with exponent larger than 1/2 can be obtained under a margin condition, which implicitly implies a faster rate of convergence. Second, it turns out that the upper bound need only depend on one term in the function Q; we call this the treatment effect term, T. For any square integrable Q, the associated treatment effect term is defined as T(X,A) ≜ Q(X,A) − E[Q(X,A)|X]. Note that d(X) ∈ arg max_a T(X, a) = arg max_a Q(X, a) a.s. Similarly, the true treatment effect term is given by

$$T_0(X,A)\triangleq Q_0(X,A)-E[Q_0(X,A)\mid X].$$ (3.2)

T₀(x, a) is the centered effect of treatment A = a at observation X = x; d₀(X) ∈ arg max_a T₀(X, a).

Theorem 3.1

Suppose p(a|x) ≥ S⁻¹ for a positive constant S for all (x, a) pairs. Assume there exists some constants C > 0 and α ≥ 0 such that

$$\mathbf{P}\left(\underset{a\in \mathcal{A}}{max}{T}_0(X,a)-\underset{a\in \mathcal{A}\backslash arg{max}_{a\in \mathcal{A}}{T}_0(X,a)}{max}{T}_0(X,a)\le \epsilon \right)\le C{\epsilon }^{\alpha }$$ (3.3)

for all positive ε. Then for any ITR d : → and square integrable function Q : × → ℝ such that d(X) ∈ arg max_a∈ Q(X, a) a.s., we have

$$V(d_0)-V(d)\le C^{\prime }{[L(Q)-L(Q_0)]}^{(1+\alpha )/(2+\alpha )},$$ (3.4)

and

$$V(d_0)-V(d)\le C^{\prime }{\left[E{(T(X,A)-T_0(X,A))}^2\right]}^{(1+\alpha )/(2+\alpha )}.$$ (3.5)

where C′ = (2^2+3αS^1+αC)^1/(2+α).

The proof of Theorem 3.1 is in Appendix A.1.

Remarks

1. We set the second maximum in (3.3) to −∞ if for an x, T₀(x, a) is constant in a and thus the set \arg max_a∈ T₀(x, a) = ∅.

2. Condition (3.3) is similar to the margin condition in classification [25, 18, 32]; in classification this assumption is often used to obtain sharp upper bounds on the excess 0 − 1 risk in terms of other surrogate risks [1]. Here can be viewed as the “margin” of T₀ at observation X = x. It measures the difference in mean responses between the optimal treatment(s) and the best suboptimal treatment(s) at x. For example, suppose X ~ U[−1, 1], P(A = 1|X) = P(A = −1|X) = 1/2 and T₀(X, A) = XA. Then the margin condition holds with C = 1/2 and α = 1. Note the margin condition does not exclude multiple optimal treatments for any observation x. However, when α > 0, it does exclude suboptimal treatments that yield a conditional mean response very close to the largest conditional mean response for a set of x with nonzero probability.

3. For C = 1, α = 0, Condition (3.3) always holds for all ε > 0; in this case (3.4) reduces to (3.1).

4. The larger the α, the larger the exponent (1 + α)/(2 + α) and thus the stronger the upper bounds in (3.4) and (3.5). However the margin condition is unlikely to hold for all ε if α is very large. An alternate margin condition and upper bound are as follows.

   Suppose p(a|x) ≥ S⁻¹ for all (x, a) pairs. Assume there is an ε > 0, such that

   $$\mathbf{P}\left(\underset{a\in \mathcal{A}}{max}{T}_0(X,a)-\underset{a\in \mathcal{A}\backslash arg{max}_{a\in \mathcal{A}}{T}_0(X,a)}{max}{T}_0(X,a)<\epsilon \right)=0.$$ (3.6)

   Then V(d₀) − V(d) ≤ 4S[L(Q) − L(Q₀)]/ε and V(d₀) − V(d) ≤ 4SE(T − T₀)²/ε.

   The proof is essentially the same as that of Theorem 3.1 and is omitted. Condition (3.6) means that T₀ evaluated at the optimal treatment(s) minus T₀ evaluated at the best suboptimal treatment(s) is bounded below by a positive constant for almost all X observations. If X assumes only a finite number of values, then this condition always holds, because we can take ε to be the smallest difference in T₀ when evaluated at the optimal treatment(s) and the suboptimal treatment(s) (note that if T₀(x, a) is constant for all a ∈ for some observation X = x, then all treatments are optimal for that observation).

5. Inequality (3.5) cannot be improved in the sense that choosing T = T₀ yields zero on both sides of the inequality. Moreover an inequality in the opposite direction is not possible, since each ITR is associated with many non-trivial T-functions. For example, suppose X ~ U[−1, 1], P(A = 1|X) = P(A = −1|X) = 1/2 and T₀(X, A) = (X − 1/3)²A. The optimal ITR is d₀(X) = 1 a.s. Consider T (X, A) = θA. Then maximizing T(X, A) yields the optimal ITR as long as θ > 0. This means that the left hand side (LHS) of (3.5) is zero, while the right hand side (RHS) is always positive no matter what value θ takes.

Theorem 3.1 supports the approach of minimizing the estimated prediction error to estimate Q₀ or T₀ and then maximizing this estimator over a ∈ to obtain an ITR. It is natural to expect that even when the approximation space used in estimating Q₀ or T₀ does not contain the truth, this approach will provide the best (highest Value) of the considered ITRs. Unfortunately this does not occur due to the mismatch between the loss functions (weighted 0–1 loss and the quadratic loss). This mismatch is indicated by remark 5 above. More precisely, note that the approximation space, say for Q₀, places implicit restrictions on the class of ITRs that will be considered. In effect the class of ITRs is = {d(X) ∈ arg max_a Q(X, a) : Q ∈ }. It turns out that minimizing the prediction error may not result in the ITR in that maximizes the Value. This occurs when the approximation space does not provide a treatment effect term close to the treatment effect term in Q₀. In the following toy example, the optimal ITR d₀ belongs to , yet the prediction error minimizer over does not yield d₀.

A toy example

Suppose X is uniformly distributed in [−1, 1], A is binary {−1, 1} with probability 1/2 each and is independent of X, and R is normally distributed with mean Q₀(X, A) = (X −1/3)²A and variance 1. It is easy to see that the optimal ITR satisfies d₀(X) = 1 a.s. and V(d₀) = 4/9. Consider approximation space = {Q(X, A; θ) = (1, X, A, XA)θ : θ ∈ ℝ⁴} for Q₀. Thus the space of ITRs under consideration is = {d(X) = sign(θ₃+θ₄X) : θ₃, θ₄ ∈ ℝ}. Note that d₀ ∈ since d₀(X) can be written as sign(θ₃ + θ₄X) for any θ₃ > 0 and θ₄ = 0. d₀ is the best treatment rule in . However, minimizing the prediction error L(Q) over yields Q^*(X, A) = (4/9−2/3X)A. The ITR associated with Q^* is d^*(X) = arg max_a∈{−1,1} Q^*(X, a) = sign(2/3 − X), which has lower Value than $d_0(V(d^{\ast })=E\left[\frac{1_{A(2/3-X)>0}R}{1/2}\right]=29/81<V(d_0))$.

4. Estimation via l₁-penalized least squares

To deal with the mismatch between minimizing the prediction error and maximizing the Value discussed in the prior section, we consider a large linear approximation space for Q₀. Since overfitting is likely (due to the potentially large number of pretreatment variables and/or large approximation space for Q₀) we use penalized least squares (see Section S.1 of the supplementary material for further discussion of the overfitting problem). Furthermore we use l₁ penalized least squares (l₁-PLS, [31]) as the l₁ penalty does some variable selection and as a result will lead to ITRs that are cheaper to implement (fewer variables to collect per patient) and easier to interpret. See Section 6 for the discussion of other potential penalization methods.

Let ${\{(X_i,A_i,R_i)\}}_{i=1}^n$ represent i.i.d. observations on n subjects in a randomized trial. For convenience, we use E_n to denote the associated empirical expectation (i.e. $E_{n}f={\sum }_{i=1}^{n}f(X_i,A_i,R_i)/n$ for any real-valued function f on × × ℝ). Let := {Q(X, A; θ) = Φ(X, A) θ, θ ∈ ℝ^J} be the approximation space for Q₀, where φ(X, A) = (φ₁(X, A), …, φ_J (X, A)) is a 1 by J vector composed of basis functions on ×, θ is a J by 1 parameter vector, and J is the number of basis functions (for clarity here J will be fixed in n, see Appendix A.2 for results with J increasing as n increases). The l₁-PLS estimator of θ is

$${\widehat{\mathit{\theta }}}_n=arg\underset{\mathit{\theta }\in {\mathbb{R}}^J}{min}\left\{E_n{[R-\mathrm{\Phi }(X,A)\mathit{\theta }]}^2+{\lambda }_n\sum _{j=1}^J{\widehat{\sigma }}_j\mid {\theta }_j\mid \right\},$$ (4.1)

where σ̂_j = [E_nφ_j(X, A)²]^1/2, θ_j is the j^th component of θ and λ_n is a tuning parameter that controls the amount of penalization. The weights σ̂_j’s are used to balance the scale of different basis functions; these weights were used in Bunea et al. [4] and van de Geer [33]. In some situations, it is natural to penalize only a subset of coefficients and/or use different weights in the penalty; see Section S.2 of the supplementary material for required modifications. The resulting estimated ITR satisfies

$${\widehat{d}}_n(X)\in arg\underset{a\in \mathcal{A}}{max}\mathrm{\Phi }(X,a){\widehat{\mathit{\theta }}}_n.$$ (4.2)

4.1. Performance guarantee for the l₁-PLS

In this section we provide finite sample upper bounds on the difference between the optimal Value and the Value obtained by the l₁-PLS estimator in terms of the prediction errors resulting from the estimation of Q₀ and T₀. These upper bounds guarantee that if Q₀ (or T₀) is consistently estimated, the estimator of d₀ will be consistent and will inherit a rate of convergence from the rate of convergence of the estimator of Q₀ (or T₀). Perhaps more importantly, the finite sample upper bounds provided below do not require the assumption that either Q₀ or T₀ is consistently estimated. Thus each upper bound includes approximation error as well as estimation error. The estimation error decreases with decreasing model sparsity and increasing sample size. An “oracle” model for Q₀ (or T₀) minimizes the sum of these two errors among suitably sparse linear models (see remark 2 after Theorem 4.3 for a precise definition of the oracle model). In finite samples, the upper bounds imply that d̂_n, the ITR produced by the l₁-PLS method, will have Value roughly as if the l₁-PLS method detects the sparsity of the oracle model and then estimates from the oracle model using ordinary least squares (see remark 3 below).

Define the prediction error minimizer θ^* ∈ ℝ^J by

$${\mathit{\theta }}^{\ast }\in arg\underset{\mathit{\theta }\in {\mathbb{R}}^J}{min}L(\mathrm{\Phi }\mathit{\theta })=arg\underset{\mathit{\theta }\in {\mathbb{R}}^J}{min}E{(R-\mathrm{\Phi }\mathit{\theta })}^2.$$ (4.3)

For expositional simplicity assume that θ^* is unique, and define the sparsity of θ ∈ ℝ^J by its l₀ norm, ||θ||₀ (see Appendix A.2 for a more general setting, where θ^* is not unique and a laxer definition of sparsity is used). As discussed above, for finite n, instead of estimating θ^*, the l₁-PLS estimator θ̂_n estimates a parameter, ${\mathit{\theta }}_n^{\ast \ast }$, possessing small prediction error but with controlled sparsity. For any bounded function f on × , let ||f||_∞ ≜ sup_x∈,a∈|f(x,a)|. ${\mathit{\theta }}_n^{\ast \ast }$ lies in the set of parameters Θ_n defined by

$${\mathrm{\Theta }}_n\triangleq \left\{\mathit{\theta }\in {\mathbb{R}}^J:{\left|\right|\mathrm{\Phi }(\mathit{\theta }-{\mathit{\theta }}^{\ast })\left|\right|}_{\infty }\le \eta ,\underset{j=1,\dots ,J}{max}\left|\frac{E[{\phi }_j\mathrm{\Phi }(\mathit{\theta }-{\mathit{\theta }}^{\ast })]}{{\sigma }_j}\right|\le 15\eta \sqrt{\frac{log(Jn)}{n}}\text{and}{\left|\right|\mathit{\theta }\left|\right|}_0\le \frac{\beta }{640U}\sqrt{\frac{n}{log(nJ)}}\right\},$$ (4.4)

where ${\sigma }_j={(E{\phi }_j^2)}^{1/2}$, and η, β and U are positive constants that will be defined in Theorem 4.1.

The first two conditions in (4.4) restrict Θ_n to θ’s with controlled distance in sup norm and with controlled distance in prediction error via first order derivatives (note that $\mid E[{\phi }_j\mathrm{\Phi }(\mathit{\theta }-{\mathit{\theta }}^{\ast })]/{\sigma }_j\mid =\mid \partial L(\mathrm{\Phi }\mathit{\theta })/\partial {\theta }_j-\partial L(\mathrm{\Phi }{\mathit{\theta }}^{\ast })/\partial {\theta }_j^{\ast }\mid /2{\sigma }_j)$. The third condition restricts Θ_n to sparse θ’s. Note that as n increases this sparsity requirement becomes laxer, ensuring that θ^* ∈ Θ_n for sufficiently large n.

When Θ_n is non-empty, ${\mathit{\theta }}_n^{\ast \ast }$ is given by

$${\mathit{\theta }}_n^{\ast \ast }=arg\underset{\mathit{\theta }\in {\mathrm{\Theta }}_n}{min}[L(\mathrm{\Phi }\mathit{\theta })+3{\left|\right|\mathit{\theta }\left|\right|}_0{\lambda }_n^2/\beta ].$$ (4.5)

Note that ${\mathit{\theta }}_n^{\ast \ast }$ is at least as sparse as θ^* since by (4.3), $L(\mathrm{\Phi }\mathit{\theta })+3{\left|\right|\mathit{\theta }\left|\right|}_0{\lambda }_n^2/\beta >L(\mathrm{\Phi }{\mathit{\theta }}^{\ast })+3{\left|\right|{\mathit{\theta }}^{\ast }\left|\right|}_0{\lambda }_n^2/\beta$ for any θ such that ||θ||₀ > ||θ^*||₀.

The following theorem provides a finite sample performance guarantee for the ITR produced by l₁-PLS method. Intuitively, this result implies that if Q₀ can be well approximated by the sparse linear representation ${\mathit{\theta }}_n^{\ast \ast }$ (so that both $L(\mathrm{\Phi }{\mathit{\theta }}_n^{\ast \ast })-L(Q_0)$ and ${\left|\right|{\mathit{\theta }}_n^{\ast \ast }\left|\right|}_0$ are small), then d̂_n will have Value close to the optimal Value in finite samples.

Theorem 4.1

Suppose p(a|x) ≥ S⁻¹ for a positive constant S for all (x, a) pairs and the margin condition (3.3) holds for some C > 0, α ≥ 0 and all positive ε. Assume

1. the error terms ε_i = R_i − Q₀(X_i, A_i), i = 1, …, n, are independent of (X_i, A_i), i …, n and are i.i.d. with E(ε_i) = 0 and $E[{\mid {\epsilon }_i\mid }^l]\le \frac{l!}{2}{c}^{l-2}{\sigma }^2$ for some c, σ² > 0 for all l ≥ 2;

2. there exist finite, positive constants U and η such that max_j=1,…,J ||φ_j||_∞/σ_j ≤ U and ||Q₀ − Φθ^*||_∞ ≤ η; and

3. E[(φ₁/σ₁, …, φ_J/σ_J )^T (φ₁/σ₁,…, φ_J/σ_J)] is positive definite, and the smallest eigenvalue is denoted by β.

Consider the estimated ITR d̂_n defined by (4.2) with tuning parameter

$${\lambda }_n\ge k\sqrt{\frac{log(Jn)}{n}},$$ (4.6)

where k = 82 max{c, σ, η}. Let Θ_n be the set defined in (4.4). Then for any n ≥ 24U² log(Jn) and for which Θ_n is non-empty, we have, with probability at least 1 − 1/n, that

$$V(d_0)-V({\widehat{d}}_n)\le C^{\prime }{\left[\underset{\mathit{\theta }\in {\mathrm{\Theta }}_n}{min}\left(L(\mathrm{\Phi }\mathit{\theta })-L(Q_0)+3{\left|\right|\mathit{\theta }\left|\right|}_0{\lambda }_n^2/\beta \right)\right]}^{\frac{1+\alpha }{2+\alpha }},$$ (4.7)

where C′ = (2^2+3αS^1+αC)^1/(2+α).

The result follows from inequality (3.4) in Theorem 3.1 and inequality (4.10) in Theorem 4.3. Similar results in a more general setting can be obtained by combining (3.4) with inequality (A.7) in Appendix A.2.

Remarks

1. Note that ${\mathit{\theta }}_n^{\ast \ast }$ is the minimizer of the upper bound on the RHS of (4.7) and that ${\mathit{\theta }}_n^{\ast \ast }$ is contained in the set { ${\mathit{\theta }}_n^{\ast ,(m)}$: m ⊂ {1, …, J}}. Each ${\mathit{\theta }}_n^{\ast ,(m)}$ satisfies ${\mathit{\theta }}_n^{\ast ,(m)}=arg{min}_{\{\mathit{\theta }\in {\mathrm{\Theta }}_n:{\theta }_j=0\text{for}\text{all}j\notin m\}}L(\mathrm{\Phi }\theta )$; that is, ${\mathit{\theta }}_n^{\ast ,(m)}$ minimizes the prediction error of the model indexed by the set m (i.e. model {Σ_j∈m φ_jθ_j : θ_j ∈ ℝ}) (within Θ_n). For each ${\mathit{\theta }}_n^{\ast ,(m)}$, the first term in the upper bound in (4.7) (i.e. $L(\mathrm{\Phi }{\mathit{\theta }}_n^{\ast ,(m)})-L(Q_0)$) is the approximation error of the model indexed by m within Θ_n. As in van de Geer [33], we call the second term $3{\left|\right|{\mathit{\theta }}_n^{\ast ,(m)}\left|\right|}_0{\lambda }_n^2/\beta$ the estimation error of the model indexed by m. To see why, first put ${\lambda }_n=k\sqrt{log(Jn)/n}$. Then, ignoring the log(n) factor, the second term is a function of the sparsity of model m relative to the sample size, n. Up to constants, the second term is a “tight” upper bound for the estimation error of the OLS estimator from model m, where “tight” means that the convergence rate in the bound is the best known rate. Note that ${\mathit{\theta }}_n^{\ast \ast }$ is the parameter that minimizes the sum of the two errors over all models. Such a model (the model corresponding to ${\mathit{\theta }}_n^{\ast \ast }$) is called an oracle model. The log(n) factor in the estimation error is the price paid for not knowing the sparsity of the oracle model. By using the l₁-PLS method, we pay by a factor of log(n) in the estimation error and as an exchange, the l₁-PLS estimator behaves roughly as if it knew the sparsity of the oracle model and as if is was estimated from the oracle model using OLS. Thus the log(n) factor can be viewed as the price paid for not knowing the sparsity of the oracle model and thus having to conduct model selection. See remark 2 after Theorem A.1 for the precise definition of the oracle model and its relationship to ${\mathit{\theta }}_n^{\ast \ast }$.

2. Suppose λ_n = o(1). Then in large samples the estimation error term $3{\left|\right|\mathit{\theta }\left|\right|}_0{\lambda }_n^2/\beta$ is negligible. In this case, ${\mathit{\theta }}_n^{\ast \ast }$ is close to θ^*. When the model Φθ^* approximates Q₀ sufficiently well, we see that setting λ_n equal to its lower bound in (4.6) provides the fastest rate of convergence of the upper bound to zero. More precisely, suppose Q₀ = Φθ^* (i.e. L(Φθ^*) − L((Q₀) = 0). Then inequality (4.7) implies that V(d₀) − V(d̂_n) ≤ O_p ((log n/n)^{(1+α)/(2+α)}). A convergence in mean result is presented in Corollary 4.1.

3. In finite samples, the estimation error $3{\left|\right|\mathit{\theta }\left|\right|}_0{\lambda }_n^2/\beta$ is nonnegligible. The argument of the minimum in the upper bound (4.7), ${\mathit{\theta }}_n^{\ast \ast }$, minimizes prediction error among parameters with controlled sparsity. In remark 2 after Theorem 4.3, we discuss how this upper bound is a tight upper bound for the OLS estimator from an oracle model in the step-wise model selection setting. In this sense, inequality (4.7) implies that decision rule produced by the l₁-PLS method will have a reduction in Value roughly as if it knew the sparsity of the oracle model and were estimated from the oracle model using OLS.

4. Assumptions 1–3 in Theorem 4.1 are employed to derive the finite sample prediction error bound for the l₁-PLS estimator θ̂_n defined in (4.1). Below we briefly discuss these assumptions.

   Assumption 1 implicitly implies that the error terms do not have heavy tails. This condition is often assumed to show that the sample mean of a variable is concentrated around its true mean with a high probability. It is easy to verify that this assumption holds if each ε_i is bounded. Moreover, it also holds for some commonly used error distributions that have unbounded support, such as the normal or double exponential.

   Assumption 2 is also used to show the concentration of the sample mean around the true mean. It is possible to replace the boundedness condition by a moment condition similar to Assumption 1. This assumption requires that all basis functions and the difference between Q₀ and its best linear approximation are bounded. Note that we do not assume to be a good approximation space for Q₀. However, if Φθ* approximates Q₀ well, η will be small, which will result in a smaller upper bound in (4.7). In fact, in the generalized result (Theorem A.1) we allow U and η to increase in n.

   Assumption 3 is employed to avoid collinearity. In fact, we only need

   $$E{[\mathrm{\Phi }({\mathit{\theta }}^{\prime }-\mathit{\theta })]}^2{\left|\right|\mathit{\theta }\left|\right|}_0\ge \beta {\left(\sum _{j\in M_0(\mathit{\theta })}{\sigma }_j\mid {\theta }_j^{\prime }-{\theta }_j\mid \right)}^2,$$ (4.8)

   for θ, θ′ belonging to a subset of ℝ^J (see Assumption A.3), where M₀(θ) ≜ {j = 1,…, J: θ_j ≠ 0}. Condition (4.8) has been used in van de Geer [33]. This condition is also similar to the restricted eigenvalue assumption in Bickel et al. [3] in which E is replaced by E_n, and a fixed design matrix is considered. Clearly, Assumption 3 is a sufficient condition for (4.8). In addition, condition (4.8) is satisfied if the correlation |Eφ_jφ_k|/(σ_jσ_k) is small for all k ∈ M₀(θ), j ≠ k and a subset of θ’s (similar results in a fixed design setting have been proved in Bickel et al. [3]. The condition on correlation is also known as “mutual coherence” condition in Bunea at al. [4]). See Bickel et al. [3] for other sufficient conditions for (4.8).

The above upper bound for V (d₀) − V (d̂_n) involves L(Φθ) − L(Q₀), which measures how well the conditional mean function Q₀ is approximated by . As we have seen in Section 3, the quality of the estimated ITR only depends on the estimator of the treatment effect term T₀. Below we provide a strengthened result in the sense that the upper bound depends only on how well we approximate the treatment effect term.

First we identify terms in the linear model that approximate T₀ (recall that T₀(X,A) ≜ Q₀(X,A) − E[Q₀(X,A)|X]). Without loss of generality, we rewrite the vector of basis functions as Φ(X, A) = (Φ⁽¹⁾(X), Φ⁽²⁾(X, A)), where Φ⁽¹⁾ = (φ₁(X), …, φ_J⁽¹⁾(X)) is composed of all components in Φ that do not contain A and Φ⁽²⁾ = (φ_J⁽¹⁾+1(X, A), …, φ_J (X, A)) is composed of all components in Φ that contain A. Since A takes only finite values and the randomization distribution p(a|x) is known, we can code A so that E[Φ⁽²⁾(X, A)^T|X] = 0 a.s. (see Section 5.2 and Appendix A.3 for examples). For any θ = (θ₁, …, θ_J)^T ∈ ℝ^J, denote θ⁽¹⁾ = (θ₁, …, θ_J⁽¹⁾)^T and θ⁽²⁾ = (θ_J⁽¹⁾+1, …, θ_J)^T. Then Φ⁽¹⁾θ⁽¹⁾ approximates E(Q₀(X, A)|X) and Φ⁽²⁾θ⁽²⁾ approximates T₀.

The following theorem implies that if the treatment effect term T₀ can be well approximated by a sparse representation, then d̂_n will have Value close to the optimal Value.

Theorem 4.2

Suppose p(a|x) ≥ S⁻¹ for a positive constant S for all (x, a) pairs and the margin condition (3.3) holds for some C > 0, α ≥ 0 and all positive ε. Assume E[Φ⁽²⁾(X, A)^T|X] = 0 a.s. Suppose Assumptions 1 – 3 in Theorem 4.1 hold. Let d̂_n be the estimated ITR with λ_n satisfying condition (4.6). Let Θ_n be the set defined in (4.4). Then for any n ≥ 24U² log(Jn) and for which Θ_n is non-empty, we have, with probability at least 1 − 1/n, that

$$V(d_0)-V({\widehat{d}}_n)\le C^{\prime }{\left[\underset{\mathit{\theta }\in {\mathrm{\Theta }}_n}{min}\left(E{({\mathrm{\Phi }}^{(2)}{\mathit{\theta }}^{(2)}-T_0)}^2+5{\left|\right|{\mathit{\theta }}^{(2)}\left|\right|}_0{\lambda }_n^2/\beta \right)\right]}^{\frac{1+\alpha }{2+\alpha }},$$ (4.9)

where C′ = (2^2+3αS^1+αC)^1/(2+α).

The result follows from inequality (3.5) in Theorem 3.1 and inequality (4.11) in Theorem 4.3.

Remarks

1. Inequality (4.9) improves inequality (4.7) in the sense that it guarantees a small reduction in Value of d̂_n as long as the treatment effect term T₀ is well approximated by a sparse linear representation; it does not require that the entire conditional mean function Q₀ be well approximated. In many situations Q₀ may be very complex, but T₀ could be very simple. This means that T₀ is much more likely to be well approximated as compared to Q₀ (indeed, if there is no difference between treatments, then T₀ ≡ 0).

2. Inequality (4.9) cannot be improved in the sense that if there is no treatment effect (i.e. T₀ ≡ 0), then both sides of the inequality are zero. This result implies that minimizing the penalized empirical prediction error indeed yields high Value (at least asymptotically) if T₀ can be well approximated.

The following asymptotic result follows from Theorem 4.2. Note that when E[Φ⁽²⁾(X, A)^T|X] = 0 a.s. (see Section 5 for examples), E(Φθ − Q₀)² = E(Φ⁽¹⁾θ⁽¹⁾ − [Q₀ − E(Q₀|X)])² + E(Φ⁽²⁾θ⁽²⁾ − T₀)². Thus the estimation of the treatment effect term T₀ is asymptotically separated from the estimation of the main effect term Q₀ − E(Q₀|X). In this case, Φ⁽²⁾θ^(2),* is the best linear approximation of the treatment effect term T₀, where θ^(2),* is the vector of components in θ^* corresponding to Φ⁽²⁾.

Corollary 4.1

Suppose p(a|x) ≥ S⁻¹ for a positive constant S for all (x, a) pairs and the margin condition (3.3) holds for some C > 0, α ≥ 0 and all positive ε. Assume E[Φ⁽²⁾(X, A)^T|X] = 0 a.s. In addition, suppose Assumptions 1 – 3 in Theorem 4.1 hold. Let d̂_n be the estimated ITR with tuning parameter ${\lambda }_n=k_1\sqrt{log(Jn)/n}$ for a constant k₁ ≥ 82 max{c, σ, η}. If T₀(X, A) = Φ⁽²⁾θ^(2),*, then

$$V(d_0)-\mathbf{E}[V({\widehat{d}}_n)]=O({(logn/n)}^{(1+\alpha )/(2+\alpha )}).$$

This result provides a guarantee on the convergence rate of V (d̂_n) to the optimal Value. More specifically, it means that if T₀ is correctly approximated, then the Value of d̂_n will converge to the optimal Value in mean at rate at least as fast as (log n/n)^{(1+α)/(2+α)} with appropriate choice of λ_n.

4.2. Prediction error bound for the l₁-PLS estimator

In this section we provide a finite sample upper bound for the prediction error of the l₁-PLS estimator θ̂_n. This result is needed to prove Theorem 4.1. Furthermore this result strengthens existing literature on l₁-PLS method in prediction. Finite sample prediction error bounds for the l₁-PLS estimator in the random design setting have been provided in Bunea et al. [4] for quadratic loss, van de Geer [33] mainly for Lipschitz loss, and Koltchinskii [15] for a variety of loss functions. With regards quadratic loss, Koltchinskii [15] requires the response Y is bounded, while both Bunea et al. [4] and van de Geer [33] assumed the existence of a sparse θ ∈ ℝ^J such that E(Φθ − Q₀)² is upper bounded by a quantity that decreases to 0 at a certain rate as n → ∞ (by permitting J to increase with n so Φ depends on n as well). We improve the results in the sense that we do not make such assumptions (see Appendix A.2 for results when Φ, J are indexed by n and J increases with n).

As in the prior sections, the sparsity of θ is measured by its l₀ norm, ||θ||₀ (see the Appendix A.2 for proofs with a laxer definition of sparsity). Recall that the parameter, ${\mathit{\theta }}_n^{\ast \ast }$ defined in (4.5) has small prediction error and controlled sparsity.

Theorem 4.3

Suppose Assumptions 1–3 in Theorem 4.1 hold. For any η₁ ≥ 0, Let θ̂_n be the l₁-PLS estimator defined by (4.1) with tuning parameter λ_n satisfying condition (4.6). Let Θ_n be the set defined in (4.4). Then for any n ≥ 24U² log(Jn) and for which Θ_n is non-empty, we have, with probability at least 1 − 1/n, that

$$L(\mathrm{\Phi }{\widehat{\mathit{\theta }}}_n)\le \underset{\mathit{\theta }\in {\mathrm{\Theta }}_n}{min}\left(L(\mathrm{\Phi }\mathit{\theta })+3{\left|\right|\mathit{\theta }\left|\right|}_0{\lambda }_n^2/\beta \right)=L(\mathrm{\Phi }{\mathit{\theta }}_n^{\ast \ast })+3{\left|\right|{\mathit{\theta }}_n^{\ast \ast }\left|\right|}_0{\lambda }_n^2/\beta .$$ (4.10)

Furthermore, suppose E[Φ⁽²⁾(X, A)^T|X] = 0 a.s. Then with probability at least 1 − 1/n,

$$E{({\mathrm{\Phi }}^{(2)}{\widehat{\mathit{\theta }}}_n^{(2)}-T_0)}^2\le \underset{\mathit{\theta }\in {\mathrm{\Theta }}_n}{min}\left(E{({\mathrm{\Phi }}^{(2)}{\mathit{\theta }}^{(2)}-T_0)}^2+5{\left|\right|{\mathit{\theta }}^{(2)}\left|\right|}_0{\lambda }_n^2/\beta \right),$$ (4.11)

The results follow from Theorem A.1 in Appendix A.2 with ρ = 0, γ= 1/8, η₁ = η₂ = η, t = log 2n and some simple algebra (notice that Assumption 3 in Theorem 4.1 is a sufficient condition for Assumptions A.3 and A.4).

Remarks

Inequality (4.11) provides a finite sample upper bound on the mean square difference between T₀ and its estimator. This result is used to prove Theorem 4.2. The remarks below discuss how inequality (4.10) contributes to the l₁-penalization literature in prediction.

1. The conclusion of Theorem 4.3 holds for all choices of λ_n that satisfy (4.6). Suppose λ_n = o(1), then $L(\mathrm{\Phi }{\mathit{\theta }}_n^{\ast \ast })-L(\mathrm{\Phi }{\mathit{\theta }}^{\ast })\to 0$ as n → ∞ (since ||θ||₀ is bounded). Then (4.10) implies that L(Φθ̂_n) − L(Φθ^*) → 0 in probability. To achieve the best rate of convergence, equal sign should be taken in (4.6).

2. Note that ${\mathit{\theta }}_n^{\ast \ast }$ minimizes $L(\mathrm{\Phi }\mathit{\theta })-L(Q_0)+3{\left|\right|\mathit{\theta }\left|\right|}_0{\lambda }_n^2/\beta$. Below we demonstrate that the minimum of $L(\mathrm{\Phi }\mathit{\theta })-L(Q_0)+3{\left|\right|\mathit{\theta }\left|\right|}_0{\lambda }_n^2/\beta$ can be viewed as the approximation error plus a “tight” upper bound of the estimation error of an “oracle” in the stepwise model selection framework (when “=” is taken in (4.6)). Here “tight” means the convergence rate in the bound is the best known rate, and “oracle” is defined as follows. Let m denote a non-empty subset of the index set {1, …, J}. Then each m represents a model which uses a non-empty subset of {φ₁, …, φ_J} as basis functions (there are 2^J − 1 such subsets). Define ${\widehat{\mathit{\theta }}}_n^{(m)}=arg{min}_{\{\mathit{\theta }\in {\mathbb{R}}^J:{\theta }_j=0\text{for}\text{all}j\notin m\}}{E}_n{(R-\mathrm{\Phi }\mathit{\theta })}^2$ and θ^*,(m) = arg min_{{θ ∈ ℝ^J:θj=0} for all _{j∉ m}}L(Φθ). In this setting, an ideal model selection criterion will pick model m^* such that $L(\mathrm{\Phi }{\widehat{\mathit{\theta }}}_n^{(m^{\ast })})={inf}_{m}L(\mathrm{\Phi }{\widehat{\mathit{\theta }}}_n^{(m)})$. ${\widehat{\mathit{\theta }}}_n^{(m^{\ast })}$ is referred as an “oracle” in Massart [20]. Note that the excess prediction error of each ${\widehat{\mathit{\theta }}}_n^{(m)}$ can be written as

   $$L(\mathrm{\Phi }{\widehat{\mathit{\theta }}}_n^{(m)})-L(Q_0)=[L(\mathrm{\Phi }{\mathit{\theta }}^{\ast ,(m)})-L(Q_0)]+[L(\mathrm{\Phi }{\widehat{\mathit{\theta }}}_n^{(m)})-L(\mathrm{\Phi }{\mathit{\theta }}^{\ast ,(m)})],$$
   where the first term is called the approximation error of model m and the second term is the estimation error. It can be shown that [2] for each model m and x_m > 0, with probability at least 1 − exp(−x_m),

   $$L(\mathrm{\Phi }{\widehat{\mathit{\theta }}}_n^{(m)})-L(\mathrm{\Phi }{\mathit{\theta }}^{\ast ,(m)})\le \mathit{constant}\times \left(\frac{x_m+\mid m\mid log(n/\mid m\mid )}{n}\right)$$
   under appropriate technical conditions, where |m| is the cardinality of the index set m. To our knowledge this is the best rate known so far. Taking x_m = log n + |m| log J and using the union bound argument, we have with probability at least 1 − O(1/n),

   $$\begin{array}{l}L({\mathrm{\Phi }}_n{\widehat{\mathit{\theta }}}_n^{(m^{\ast })})-L(Q_0)\\ =\underset{m}{min}\left([L(\mathrm{\Phi }{\mathit{\theta }}^{\ast ,(m)})-L(Q_0)]+L(\mathrm{\Phi }{\widehat{\mathit{\theta }}}_n^{(m)})-L(\mathrm{\Phi }{\mathit{\theta }}^{\ast ,(m)})\right)\\ \le \underset{m}{min}\left([L(\mathrm{\Phi }{\mathit{\theta }}^{\ast ,(m)})-L(Q_0)]+\mathit{constant}\times \frac{\mid m\mid log(Jn)}{n}\right)\\ =\underset{\mathit{\theta }}{min}\left([L(\mathrm{\Phi }\mathit{\theta })-L(Q_0)]+\mathit{constant}\times \frac{{\left|\right|\mathit{\theta }\left|\right|}_{0}log(Jn)}{n}\right).\end{array}$$ (4.12)
   On the other hand, take λ_n so that condition (4.6) holds with “=”. (4.10) implies that, with probability at least 1 − 1/n,

   $$L(\mathrm{\Phi }{\widehat{\mathit{\theta }}}_n)-L(Q_0)\le \underset{\mathit{\theta }\in {\mathrm{\Theta }}_n}{min}\left([L(\mathrm{\Phi }\mathit{\theta })-L(Q_0)]+\mathit{constant}\times \frac{{\left|\right|\mathit{\theta }\left|\right|}_{0}log(Jn)}{n}\right),$$

   which is essentially (4.12) with the constraint of θ ∈ Θ_n. (The “constant” in the above inequalities may take different values.) Since $\mathit{\theta }={\mathit{\theta }}_n^{\ast \ast }$ minimizes the approximation error plus a tight upper bound for the estimation error in the oracle model, within θ ∈ Θ_n, we refer to ${\mathit{\theta }}_n^{\ast \ast }$ as an oracle.

3. The result can be used to emphasize that l₁ penalty behaves similarly as the l₀ penalty. Note that θ̂_n minimizes the empirical prediction error, E_n(R − Θθ)², plus an l₁ penalty whereas θ^**(u_n) minimizes the prediction error L(Φθ) plus an l₀ penalty. We provide an intuitive connection between these two quantities. First note that E_n(R − Φθ)² estimates L(Φθ) and σ̂_j estimates σ_j. We use “≈” to denote this relationship. Thus

   $$E_n{(R-\mathrm{\Phi }\mathit{\theta })}^2+{\lambda }_n\sum _{j=1}^J{\widehat{\sigma }}_j\mid {\theta }_i\mid \approx L(\mathrm{\Phi }\mathit{\theta })+{\lambda }_n\sum _{j=1}^J{\sigma }_j\mid {\theta }_j\mid \le L(\mathrm{\Phi }\mathit{\theta })+{\lambda }_n\sum _{j=1}^J{\sigma }_j\mid {\widehat{\theta }}_{n,j}-{\theta }_j\mid +{\lambda }_n\sum _{j=1}^J{\sigma }_j\mid {\widehat{\theta }}_{n,j}\mid ,$$ (4.13)

   where θ̂_n,j is the j_th component of θ̂_n. In Appendix B we show that for any θ ∈ Θ_n, ${\lambda }_n{\sum }_{j=1}^J{\sigma }_j\mid {\widehat{\theta }}_{n,j}-{\theta }_j\mid$ is upper bounded by ${\left|\right|\mathit{\theta }\left|\right|}_0{\lambda }_n^2/\beta$ up to a constant with a high probability. Thus θ̂_n minimizes (4.13) and θ^**(u_n) roughly minimizes an upper bound of (4.13).

4. The constants involved in the theorem can be improved; we focused on readability as opposed to providing the best constants.

5. A Practical Implementation and an Evaluation

In this section we develop a practical implementation of the l₁-PLS method, compare this method to two commonly used alternatives and lastly illustrate the method using the motivating data from the Nefazodone-CBASP trial [13].

A realistic implementation of l₁-PLS method should use a data-dependent method to select the tuning parameter, λ_n. Since the primary goal is to maximize the Value, we select λ_n to maximize a cross validated Value estimator. For any ITR d, it is easy to verify that E[(R − V (d))1_A=d(X)/p(A|X)] = 0. Thus an unbiased estimator of V (d) is

$$E_n[1_{A=d(X)}R/p(A\mid X)]/E_n[1_{A=d(X)}/p(A\mid X)]$$

[21] (recall that the randomization distribution p(a|X) is known). We split the data into 10 roughly equal-sized parts; then for each λ_n we apply the l₁-PLS based method on each 9 parts of the data to obtain an ITR, and estimate the Value of this ITR using the remaining part; the λ_n that maximizes the average of the 10 estimated Values is selected. Since the Value of an ITR is noncontinuous in the parameters, this usually results in a set of candidate λ_n’s achieving maximal Value. In the simulations below the resulting λ_n is nonunique in around 97% of the data sets. If necessary, as a second step we reduce the set of λ_n’s by including only λ_n’s leading to the ITR’s using the least number of variables. In the simulations below this second criterion effectively reduced the number of candidate λ_n’s in around 25% of the data sets, however multiple λ_n’s still remained in around 90% of the data sets. This is not surprising since the Value of an ITR only depends on the relative magnitudes of parameters in the ITR. In the third step we select the λ_n that minimizes the 10-fold cross validated prediction error estimator from the remaining candidate λ_n’s; that is, minimization of the empirical prediction error is used as a final tie breaker.

5.1. Simulations

A first alternative to l₁-PLS is to use ordinary least squares (OLS). The estimated ITR is d̂_OLS ∈ arg max_a Φ(X, a)θ̂_OLS where θ̂_OLS is the OLS estimator of θ. A second alternative is called “prognosis prediction” [14]. Usually this method employees multiple data sets, each of which involves one active treatment. Then the treatment that is associated with the best predicted prognosis [14] is selected. We implement this method by estimating E(R|X, A = a) via least squares with l₁ penalization for each treatment group (each a ∈ ) separately. The tuning parameter involved in each treatment group is selected by minimizing the 10-fold cross-validated prediction error estimator. The resulting ITR satisfies d̂_PP (X) ∈ arg max_a∈ Ê(R|X, A = a) where the subscript “PP” denotes prognosis prediction.

For simplicity we consider binary A. All three methods use the same number of data points and the same number of basis functions but use these data points/basis functions differently. l₁-PLS and OLS use all J basis functions to conduct estimation with all n data points whereas the prognosis prediction method splits the data into the two treatment groups and uses J/2 basis functions to conduct estimation with the n/2 data points in each of the two treatment groups. To ensure the comparison is fair across the three methods, the approximation model for each treatment group is consistent with the approximation model used in both l₁-PLS and OLS (e.g. if Q₀ is approximated by (1, X, A, XA)θ in l₁-PLS and OLS, then in prognosis prediction we approximate E(R|X, A = a) by (1, X)θ_PP for each treatment group). We do not penalize the intercept coefficient in either prognosis prediction or l₁-PLS.

The three methods are compared using two criteria: 1) Value maximization; and 2) simplicity of the estimated ITRs (measured by the number of variables/basis functions used in the rule).

We illustrate the comparison of the three methods using 4 examples selected to reflect three scenarios; please see Section S.3 of the supplementary material for 4 further examples.

1. There is no treatment effect (i.e. Q₀ is constructed so that T₀ = 0; example 1). In this case, all ITRs yield the same Value. Thus the simplest rule is preferred.

2. There is a treatment effect and the treatment effect term T₀ is correctly modeled (example 4 for large n, and example 2). In this case, minimizing the prediction error will yield the ITR that maximizes the Value.

3. There is a treatment effect and the treatment effect term T₀ is misspecified (example 4 for small n, and example 3). In this case, there might be a mismatch between prediction error minimization and Value maximization.

The examples are generated as follows. The treatment A is generated uniformly from {−1, 1} independent of X and the response R. The response R is normally distributed with mean Q₀(X, A). In examples 1–3, X ~ U [−1, 1]⁵ and we consider three simple examples for Q₀. In example 4, X ~ U [0, 1] and we use a complex Q₀, where Q₀(X, 1) and Q(X, −1) are similar to the blocks function used in Donoho and Johnstone [8]. Further details of the simulation design are provided in Appendix A.3.

We consider two types of approximation models for Q₀. In examples 1–3, we approximate Q₀ by (1, X, A, XA)θ. In example 4, we approximate Q₀ by Haar wavelets. The number of basis functions may increase as n increases (we index J, Φ and θ^* by n in this case). Plots for Q₀(X, A) and the associated best wavelet fits ${\mathrm{\Phi }}_n(X,A){\mathit{\theta }}_n^{\ast }$ are provided in Figure 1.

Fig 1.

Plots for: the conditional mean function Q₀(X; A) (left), Q₀(X; A) and the associated best wavelet fit when J_n = 8 (middle), and Q₀(X; A) and the associated best wavelet fit when J_n = 128 (right) (example 4).

For each example, we simulate data sets of sizes n = 2^k for k = 5, …, 10. 1000 data sets are generated for each sample size. The Value of each estimated ITR is evaluated via Monte Carlo using a test set of size 10, 000. The Value of the optimal ITR is also evaluated using the test set.

Simulation results are presented in Figure 2. When the approximation model is of high quality, all methods produce ITRs with similar Value (see examples 1, 2 and example 4 for large n). However, when the approximation model is poor, the l₁-PLS method may produce highest Value (see example 3). Note that in example 3 settings in which the sample size is small, the Value of the ITR produced by l₁-PLS method has larger median absolute deviation (MAD) than the other two methods. One possible reason is that due to the mismatch between maximizing the Value and minimizing the prediction error, the Value estimator plays a strong role in selecting λ_n. The non-smoothness of the Value estimator combined with the mismatch results in very different λ_ns and thus the estimated decision rules vary greatly from data set to data set in this example. Nonetheless, the l₁-PLS method is still preferred after taking the variation into account; indeed l₁-PLS produces ITRs with higher Value than both OLS and PP in around 46%, 55% and 67% in data sets of sizes n = 32, 64 and 128, respectively. Furthermore, in general the l₁-PLS method uses much fewer variables for treatment assignment than the other two methods. This is expected because the OLS method does not have variable selection functionality and the PP method will use all variables that are predictive of the response R whereas the use of the Value in selecting the tuning parameter in l₁-PLS discounts variables that are only useful in predicting the response (and less useful in selecting the best treatment).

Fig 2.

Comparison of the l₁-PLS based method with the OLS method and the PP method (examples 1 – 4): Plots for medians and median absolute deviations (MAD) of the Value of the estimated decision rules (top panels) and the number of variables (terms) needed for treatment assignment (including the main treatment effect term, bottom panels) over 1000 samples versus sample size on the log scale. The black dash-dotted line in each plot on the first row denotes the Value of the optimal treatment rule, V (d₀), for each example. (n = 32; 64; 128; 256; 512; 1024. The corresponding numbers of basis functions in example 4 are J_n = 8; 16; 32; 64; 64; 128).

5.2. Nefazodone-CBASP trial example

The Nefazodone-CBASP trial was conducted to compare the efficacy of several alternate treatments for patients with chronic depression. The study randomized 681 patients with non-psychotic chronic major depressive disorder (MDD) to either Nefazodone, cognitive behavioral-analysis system of psychotherapy (CBASP) or the combination of the two treatments. Various assessments were taken throughout the study, among which the score on the 24-item Hamilton Rating Scale for Depression (HRSD) was the primary outcome. Low HRSD scores are desirable. See Keller et al. [13] for more detail of the study design and the primary analysis.

In the data analysis, we use a subset of the Nefazodone-CBASP data consisting of 656 patients for whom the response HRSD score was observed. In this trial, pairwise comparisons show that the combination treatment resulted in significantly lower HRSD scores than either of the single treatments. There was no overall difference between the single treatments.

We use l₁-PLS to develop an ITR. In the analysis the HRSD score is reverse coded so that higher is better. We consider 50 pretreatment variables X = (X₁,…, X₅₀). Treatments are coded using contrast coding of dummy variables A = (A₁, A₂), where A₁ = 2 if the combination treatment is assigned and −1 otherwise and A₂ = 1 if CBASP is assigned, −1 if nefazodone and 0 otherwise. The vector of basis functions, Φ(X, A), is of the form (1, X, A₁, XA₁, A₂, XA₂). So the number of basis functions is J = 153. As a contrast, we also consider the OLS method and the PP method (separate prognosis prediction for each treatment). The vector of basis functions used in PP is (1, X) for each of the three treatment groups. Neither the intercept term nor the main treatment effect terms in l₁-PLS or PP is penalized (see Section S.2 of the supplementary material for the modification of the weights σ̂_j used in (4.1)).

The ITR given by the l₁-PLS method recommends the combination treatment to all (so none of the pretreatment variables enter the rule). On the other hand, the PP method produces an ITR that uses 29 variables. If the rule produced by PP were used to assign treatment for the 656 patients in the trial, it would recommend the combination treatment for 614 patients and nefazodone for the other 42 patients. In addition, the OLS method will use all the 50 variables. If the ITR produced by OLS were used to assign treatment for the 656 patients in the trial, it would recommend the combination treatment for 429 patients, nefazodone for the 145 patients and CBASP for the other 82 patients.

6. Discussion

Our goal is to construct a high quality ITR that will benefit future patients. We considered an l₁-PLS based method and provided a finite sample upper bound for V (d₀) − V (d̂_n), the excess Value of the estimated ITR.

The use of an l₁ penalty allows us to consider a large model for the conditional mean function Q₀ yet permits a sparse estimated ITR. In fact, many other penalization methods such as SCAD [9] and l₁ penalty with adaptive weights (adaptive Lasso; [37]) also have this property. We choose the non-adaptive l₁ penalty to represent these methods. Interested readers may justify other PLS methods using similar proof techniques.

The high probability finite sample upper bounds (i.e. (4.7) and (4.9)) cannot be used to construct a prediction/confidence interval for V (d₀) − V (d̂_n) due to the unknown quantities in the bound. How to develop a tight computable upper bound to assess the quality of d̂_n is an open question.

We used cross validation with Value maximization to select the tuning parameter involved in the l₁-PLS method. As compared to the OLS method and the PP method, this method may yield higher Value when T₀ is misspecified. However, since only the Value is used to select the tuning parameter, this method may produce a complex ITR for which the Value is only slightly higher than that of a much simpler ITR. In this case, a simpler rule may be preferred due to the interpretability and cost of collecting the variables. Investigation of a tuning parameter selection criterion that trades off the Value with the number of variables in an ITR is needed.

This paper studied a one stage decision problem. However, it is evident that some diseases require time-varying treatment. For example, individuals with a chronic disease often experience a waxing and waning course of illness. In these settings the goal is to construct a sequence of ITRs that tailor the type and dosage of treatment through time according to an individual’s changing status. There is an abundance of statistical literature in this area [29, 30, 22, 23, 26, 17, 34, 35]. Extension of the least squares based method to the multi-stage decision problem has been presented in Murphy [23]. The performance of l₁ penalization in this setting is unclear and worth investigation.

APPENDIX

A.1. Proof of Theorem 3.1

For any ITR d: → , denote ΔT_d(X) ≜ max_a∈T₀(X, a) − T₀(X, d(X)). Using similar arguments to that in Section 2, we have V (d₀) − V (d) = E(ΔT_d). If V (d₀) − V (d) = 0, then (3.4) and (3.5) automatically hold. Otherwise, E(ΔT_d)² ≥ (EΔT_d)² > 0. In this case, for any ε > 0, define the event

$${\mathrm{\Omega }}_{\epsilon }=\left\{\underset{a\in \mathcal{A}}{max}{T}_0(X,a)-\underset{a\in \mathcal{A}\backslash arg{max}_{a\in \mathcal{A}}{T}_0(X,a)}{max}{T}_0(X,a)\le \epsilon \right\}.$$

Then ΔT_d ≤ (ΔT_d)²/ε on the event ${\mathrm{\Omega }}_{\epsilon }^C$. This together with the fact that ΔT_d ≤ (ΔT_d)²/ε + ε/4 implies

$$\begin{array}{l}V(d_0)-V(d)=E(1_{{\mathrm{\Omega }}_{\epsilon }^C}\mathrm{\Delta }{T}_d)+E(1_{{\mathrm{\Omega }}_{\epsilon }}\mathrm{\Delta }{T}_d)\\ \le \frac{1}{\epsilon }E[1_{{\mathrm{\Omega }}_{\epsilon }^C}{(\mathrm{\Delta }{T}_d)}^2]+E\left[1_{{\mathrm{\Omega }}_{\epsilon }}\left(\frac{{(\mathrm{\Delta }{T}_d)}^2}{\epsilon }+\frac{\epsilon }{4}\right)\right]\\ =\frac{1}{\epsilon }E[{(\mathrm{\Delta }{T}_d)}^2]+\frac{\epsilon }{4}P({\mathrm{\Omega }}_{\epsilon })\le \frac{1}{\epsilon }E[{(\mathrm{\Delta }{T}_d)}^2]+\frac{C}{4}{\epsilon }^{1+\alpha },\end{array}$$

where the last inequality follows from the margin condition (3.3). Choosing ε = (4E(ΔT_d)²/C)^1/(2+α) to minimize the above upper bound yields

$$V(d_0)-V(d)\le 2^{\alpha /(2+\alpha )}{C}^{1/(2+\alpha )}{[E{(\mathrm{\Delta }{T}_d)}^2]}^{(1+\alpha )/(2+\alpha )}.$$ (A.1)

Next, for any d and Q such that d(X) ∈ max_a∈ Q(X, a) and decomposition Q(X, A) into W(X) + T(X, A),

$$\begin{array}{l}E{(\mathrm{\Delta }{T}_d)}^2=E\left[{\left(\underset{a\in \mathcal{A}}{max}{T}_0(X,a)-\underset{a\in \mathcal{A}}{max}T(X,a)+T(X,d(X))-T_0(X,d(X))\right)}^2\right]\\ \le 2E\left[{\left(\underset{a\in \mathcal{A}}{max}{T}_0(X,a)-\underset{a\in \mathcal{A}}{max}T(X,a)\right)}^2+{\left(T(X,d(X))-T_0(X,d(X))\right)}^2\right]\\ \le 4E\left[\underset{a\in \mathcal{A}}{max}{(T(X,a)-T_0(X,a))}^2\right],\end{array}$$

where the last inequality follows from the fact that neither |max_a T₀(X, a)− max_a T (X, a)| nor |T (X, d(X))−T₀(X, d(X))| is larger than max_a |T (X, a)− T₀(X, a)|. Since p(a|x) ≥ S⁻¹ for all (x, a) pairs, we have

$$E{(\mathrm{\Delta }{T}_d)}^2\le 4SE\left[\sum _{a\in \mathcal{A}}{(T(X,a)-T_0(X,a))}^{2}p(a\mid X)\right]=4SE{(T(X,A)-T_0(X,A))}^2.$$ (A.2)

Inequality (3.5) follows by substituting (A.2) into (A.1) and setting W(X, A) = E[Q(X, A)|X]. Inequality (3.4) follows by setting W(X) = 0 and noticing that ΔT_d(X) = max_a∈ Q₀(X, a) − Q₀(X, d(X)).

A.2. Generalization of Theorem 4.3

In this section, we present a generalization of Theorem 4.3 where J may depend on n and the sparsity of any θ ∈ ℝ^J is measured by the number of “large” components in θ as described in Zhang and Huang [36]. In this case, J, F and the prediction error minimizer θ^* are denoted as J_n, Φ_n and ${\mathit{\theta }}_n^{\ast }$, respectively. All relevant quantities and assumptions are re-stated below.

Let |M| denote the cardinality of any index set M ⊆ {1,…, J_n}. For any θ ∈ ℝ^J_n and constant ρ ≥ 0, define

$$M_{\rho {\lambda }_n}(\mathit{\theta })\in arg\underset{\{M\subseteq \{1,\dots ,J_n\}:{\sum }_{j\in \{1,\dots ,J_n\}\backslash M}{\sigma }_j\mid {\theta }_j\mid \le \rho \mid M\mid {\lambda }_n\}}{min}\mid M\mid .$$

Then M_ρλn (θ) is the smallest index set that contains only “large” components in θ. |M_ρλn (θ)| measures the sparsity of θ. It is easy to see that when ρ = 0, M₀(θ) is the index set of nonzero components in θ and |M₀(θ)| = ||θ||₀. Moreover, M_ρλn (θ) is an empty set if and only if θ = 0.

Let [ ${\mathit{\theta }}_n^{\ast }$] be the set of most sparse prediction error minimizers in the linear model, i.e.

$$[{\mathit{\theta }}_n^{\ast }]=arg\underset{\mathit{\theta }\in arg{min}_{\mathit{\theta }}L({\mathrm{\Phi }}_n\mathit{\theta })}{min}\mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid .$$ (A.3)

Note that [ ${\mathit{\theta }}_n^{\ast }$] depends on ρλ_n.

To derive the finite sample upper bound for L(Φ_nθ̂_n), we need the following assumptions.

Assumption A.1

The error terms ε_i, i = 1,…, n are independent of (X_i, A_i), i = 1,…, n and are i.i.d. with E(ε_i) = 0 and $E[{\mid {\epsilon }_i\mid }^l]\le \frac{l!}{2}{c}^{l-2}{\sigma }^2$ for some c, σ² > 0 for all l ≥ 2.

Assumption A.2

For all n ≥ 1,

1. there exists an 1 ≤ U_n < ∞ such that max_j=1,…,Jn ||φ_j||_∞/σ_j ≤ U_n, where ${\sigma }_j\triangleq {(E{\phi }_j^2)}^{1/2}$.

2. there exists an 0 < η_1,n < ∞, such that ${sup}_{\mathit{\theta }\in [{\mathit{\theta }}_n^{\ast }]}{\left|\right|Q_0-{\mathrm{\Phi }}_n\mathit{\theta }\left|\right|}_{\infty }\le {\eta }_{1,n}$.

For any 0 ≤ γ < 1/2, η_2,n ≥ 0 (which may depend on n) and tuning parameter λ_n, define

$${\mathrm{\Theta }}_n^o=\{\mathit{\theta }\in {\mathbb{R}}^{J_n}:\exists {\mathit{\theta }}^o\in [{\mathit{\theta }}_n^{\ast }]\text{s}.\text{t}.{\left|\right|{\mathrm{\Phi }}_n(\mathit{\theta }-{\mathit{\theta }}^o)\left|\right|}_{\infty }\le {\eta }_{2,n}\text{and}\underset{j=1,\dots ,J_n}{max}\left|E\left[{\mathrm{\Phi }}_n(\mathit{\theta }-{\mathit{\theta }}^o)\frac{{\phi }_j}{{\sigma }_j}\right]\right|\le \gamma {\lambda }_n\}.$$

Assumption A.3

For any n ≥ 1, there exists a β_n > 0 such that

$$E{[{\mathrm{\Phi }}_n(\stackrel{\sim }{\mathit{\theta }}-\mathit{\theta })]}^2\mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid \ge {\beta }_n\left[{\left(\sum _{j\in M_{\rho {\lambda }_n}(\mathit{\theta })}{\sigma }_j\mid {\stackrel{\sim }{\theta }}_j-{\theta }_j\mid \right)}^2-{\rho }^2{\mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid }^2{\lambda }_n^2\right]$$

for all $\mathit{\theta }\in {\mathrm{\Theta }}_n^o\backslash \{\mathbf{0}\}$, θ̃ ∈ ℝ^J_n and ${\sum }_{j\in \{1,\dots ,J_n\}\backslash M_{\rho {\lambda }_n}(\mathit{\theta })}{\sigma }_j\mid {\stackrel{\sim }{\theta }}_j\mid \le \frac{2\gamma +5}{1-2\gamma }({\sum }_{j\in M_{\rho {\lambda }_n}(\mathit{\theta })}\mid {\stackrel{\sim }{\theta }}_j-{\theta }_j\mid +\rho \mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid {\lambda }_n)$.

When $E({\mathrm{\Phi }}_n^{(2)}{(X,A)}^T\mid X)=\mathbf{0}$ a.s. ( ${\mathrm{\Phi }}_n^{(2)}$ is defined in Section 4.1), we need an extra assumption to derive the finite sample upper bound for the mean square error of the treatment effect estimator, $E[{{\mathrm{\Phi }}_n^{(2)}{\widehat{\mathit{\theta }}}_n^{(2)}-T_0(X,A)]}^2$ (recall that T₀(X,A) ≜ Q₀(X, A) − E[Q₀(X,A)|X]).

Assumption A.4

For any n ≥ 1, there exists a β_n > 0 such that

$$E{[{\mathrm{\Phi }}_n^{(2)}({\stackrel{\sim }{\mathit{\theta }}}^{(2)}-{\mathit{\theta }}^{(2)})]}^2\mid M_{\rho {\lambda }_n}^{(2)}(\mathit{\theta })\mid \ge {\beta }_n\left[{\left(\sum _{j\in M_{\rho {\lambda }_n}^{(2)}(\mathit{\theta })}{\sigma }_j\mid {\stackrel{\sim }{\theta }}_j-{\theta }_j\mid \right)}^2-{\rho }^2{\mid M_{\rho {\lambda }_n}^{(2)}(\mathit{\theta })\mid }^2{\lambda }_n^2\right]$$

for all $\mathit{\theta }\in {\mathrm{\Theta }}_n^o\backslash \{\mathbf{0}\}$, θ̃ ∈ ℝ^J_n and ${\sum }_{j\in \{1,\dots ,J_n\}\backslash M_{\rho {\lambda }_n}(\mathit{\theta })}{\sigma }_j\mid {\stackrel{\sim }{\theta }}_j\mid \le \frac{2\gamma +5}{1-2\gamma }({\sum }_{j\in M_{\rho {\lambda }_n}(\mathit{\theta })}\mid {\stackrel{\sim }{\theta }}_j-{\theta }_j\mid +\rho \mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid {\lambda }_n)$, where

$$M_{\rho {\lambda }_n}^{(2)}(\mathit{\theta })\in arg\underset{\{M\subseteq \{J_n^{(1)}+1,\dots ,J_n\}:{\sum }_{j\in (J_n^{(1)}+1,\dots ,J_n)\backslash M}{\sigma }_j\mid {\theta }_j\mid \le \rho \mid M\mid {\lambda }_n\}}{min}\mid M\mid$$

is the smallest index set that contains only large components in θ⁽²⁾.

Note that here for simplicity, we assume that Assumptions A.3 and A.4 hold with the same value of β_n. And with out loss of generality, we can always choose a small enough β_n so that ρβ_n ≤ 1 for a given ρ.

For any t > 0, define

$${\mathrm{\Theta }}_n=\{\mathit{\theta }\in {\mathrm{\Theta }}_n^o:\mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid \le \frac{{(1-2\gamma )}^2{\beta }_n}{120}\left[\sqrt{\frac{1}{9}+\frac{n}{2U_n^2[log(3J_n(J_n+1))+t]}}-\frac{1}{3}\right]\}.$$ (A.4)

Note that we allow U_n, η_1,n, η_2,n and ${\beta }_n^{-1}$ to increase as n increases. However, if those quantities are small, the upper bound in (A.7) will be tighter.

Theorem A.1

Suppose Assumptions A.1 and A.2 hold. For any given 0 ≤ γ < 1/2, η_2,n > 0, ρ ≥ 0 and t > 0, let θ̂_n be the l₁-PLS estimator defined in (4.1) with tuning parameter

$${\lambda }_n\ge \frac{8max\{3c,2({\eta }_{1,n}+{\eta }_{2,n})\}{U}_n(log6J_n+t)}{(1-2\gamma )n}+\frac{12max\{\sigma ,({\eta }_{1,n}+{\eta }_{2,n})\}}{(1-2\gamma )}\sqrt{\frac{2(log6J_n+t)}{n}}.$$ (A.5)

Suppose Assumption A.3 holds with ρβ_n ≤ 1. Let Θ_n be the set defined in (A.4) and assume Θ_n is non-empty. If

$$\frac{log2J_n}{n}\le \frac{2{(1-2\gamma )}^2}{27U_n^2-10\gamma -22},$$ (A.6)

then with probability at least $1-exp(-k_n^{\prime }n)-exp(-t)$, we have

$$L({\mathrm{\Phi }}_n{\widehat{\mathit{\theta }}}_n)\le \underset{\mathit{\theta }\in {\mathrm{\Theta }}_n}{min}\left[L({\mathrm{\Phi }}_n\mathit{\theta })+K_n\frac{\mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid }{{\beta }_n}{\lambda }_n^2\right],$$ (A.7)

where $k_n^{\prime }=13{(1-2\gamma )}^2[6(27U_n^2-10\gamma -22)]$ and K_n = [40γ(12β_nρ + 2γ + 5)]/[(1 − 2γ)(2γ + 19)] + 130(12β_nρ + 2γ + 5)²/[9(2γ + 19)²].

Furthermore, suppose $E({\mathrm{\Phi }}_n^{(2)}{(X,A)}^T\mid X)=\mathbf{0}$ a.s. If Assumption A.4 holds with ρβ_n ≤ 1, then with probability at least $1-exp(-k_n^{\prime }n)-exp(-t)$, we have

$$E{({\mathrm{\Phi }}_n^{(2)}{\widehat{\mathit{\theta }}}_n^{(2)}-T_0)}^2\le \underset{\mathit{\theta }\in {\mathrm{\Theta }}_n}{min}\left[E{({\mathrm{\Phi }}_n^{(2)}{\mathit{\theta }}^{(2)}-T_0)}^2+K_n^{\prime }\frac{\mid M_{\rho {\lambda }_n}^2(\mathit{\theta })\mid }{{\beta }_n}{\lambda }_n^2\right].$$

where $K_n^{\prime }=20(12{\beta }_n\rho +2\gamma +5)\{\gamma /[(1-2\gamma )(7-6{\beta }_n\rho )]+[3(1-2\gamma ){\beta }_n\rho +10(2\gamma +5)]/[9{(2\gamma +19)}^2]\}$.

Remark

1. Note that K_n is upper bounded by a constant under the assumption β_nρ ≤ 1. In the asymptotic setting when n → ∞ and J_n → ∞, (A.7) implies that with probability tending to 1, $L({\mathrm{\Phi }}_n{\widehat{\mathit{\theta }}}_n)-L({\mathrm{\Phi }}_n{\mathit{\theta }}_n^{\ast })\to 0$ if (i) $\mid M_{\rho {\lambda }_n}({\mathit{\theta }}_n^{\ast })\mid {\lambda }_n^2/{\beta }_n=o(1)$, (ii) $U_n^{2}log{J}_n/n\le k_1$ and $\mid M_{\rho {\lambda }_n}({\mathit{\theta }}_n^{\ast })\mid \le k_2{\beta }_n\sqrt{n/(U_n^{2}log{J}_n)}$ for some sufficiently small positive constants k₁ and k₂, and (iii) ${\lambda }_n\ge k_{3}max\{1,{\eta }_{1,n}+{\eta }_{2,n}\}\sqrt{log{J}_n/n}$ for a sufficiently large constant k₃, where ${\mathit{\theta }}_n^{\ast }\in [{\mathit{\theta }}_n^{\ast }]$ (take t = log J_n).

2. Below we briefly discuss Assumptions A.2 – A.4.

   Assumption A.2 is very similar to Assumption 2 in Theorem 4.1 (which is used to prove the concentration of the sample mean around the true mean), except that U_n and η_1,n may increase as n increases. This relaxation allows the use of basis functions for which the sup norm max_j ||φ_j||_∞ is increasing in n (e.g. the wavelet basis used in example 4 of the simulation studies).

   Assumption A.3 is a generalization of condition (4.8) (which has been discussed in remark 4 following Theorem Theorem 4.1)) to the case where J_n may increase in n and the sparsity of a parameter is measured by the number of “large” components as described at the beginning of this section. This condition is used to avoid the collinearity problem. It is easy to see that when ρ = 0 and β_n is fixed in n, this assumption simplifies to condition (4.8).

   Assumption A.4 puts a strengthened constraint on the linear model of the treatment effect part, as compared to Assumption A.3. This assumption, together with Assumption A.3, is needed in deriving the upper bound for the mean square error of the treatment effect estimator. It is easy to verify that if $E[{\mathrm{\Phi }}_n^T{\mathrm{\Phi }}_n]$ is positive definite, then both A.3 and A.4 hold. Although the result is about the treatment effect part, which is asymptotically independent of the main effect of X (when $E[{\mathrm{\Phi }}_n^{(2)}(X,A)\mid X]=\mathbf{0}$ a.s.), we still need Assumption A.3 to show that the cross product term $E_n[({\mathrm{\Phi }}_n^{(1)}{\widehat{\mathit{\theta }}}_n^{(1)}-{\mathrm{\Phi }}_n^{(1)}{\mathit{\theta }}^{(1)})({\mathrm{\Phi }}_n^{(2)}{\widehat{\mathit{\theta }}}_n^{(2)}-{\mathrm{\Phi }}_n^{(2)}{\mathit{\theta }}^{(2)})]$ is upper bounded by a quantity converging to 0 at the desired rate. We may use a really poor model for the main effect part E(Q₀(X, A)|X) (e.g. ${\mathrm{\Phi }}_n^{(1)}\equiv 1$), and Assumption A.4 implies Assumption A.3 when ρ = 0. This poor model only effects the constants involved in the result. When the sample size is large (so that λ_n is small), the estimated ITR will be of high quality as long as T₀ is well approximated.

Proof

For any θ ∈ Θ_n, define the events

$$\begin{array}{l}{\mathrm{\Omega }}_1=\underset{j=1}{\overset{J_n}{\cap }}\left\{\frac{2(1+\gamma )}{3}{\sigma }_j\le {\widehat{\sigma }}_j\le \frac{2(2-\gamma )}{3}{\sigma }_j\right\}(\text{where}{\widehat{\sigma }}_j\triangleq {(E_n{\phi }_j^2)}^{1/2}),\\ {\mathrm{\Omega }}_2(\mathit{\theta })=\left\{\underset{j,k=1,\dots ,J_n}{max}\left|(E-E_n)\left(\frac{{\phi }_j{\phi }_k}{{\sigma }_j{\sigma }_k}\right)\right|\le \frac{{(1-2\gamma )}^2{\beta }_n}{120\mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid }\right\},\\ {\mathrm{\Omega }}_3(\mathit{\theta })=\left\{\underset{j=1,\dots ,J_n}{max}\left|E_n\left[(R-{\mathrm{\Phi }}_n\mathit{\theta })\frac{{\phi }_j}{{\sigma }_j}\right]\right|\le \frac{4\gamma +1}{6}{\lambda }_n\right\}.\end{array}$$

Then there exists a ${\mathit{\theta }}^o\in [{\mathit{\theta }}_n^{\ast }]$ such that

$$\begin{array}{l}L({\mathrm{\Phi }}_n{\widehat{\mathit{\theta }}}_n)=L({\mathrm{\Phi }}_n\mathit{\theta })+2E[({\mathrm{\Phi }}_n{\mathit{\theta }}^o-{\mathrm{\Phi }}_n\mathit{\theta }){\mathrm{\Phi }}_n(\mathit{\theta }-{\widehat{\mathit{\theta }}}_n)]+E{[{\mathrm{\Phi }}_n({\widehat{\mathit{\theta }}}_n-\mathit{\theta })]}^2\\ \le L({\mathrm{\Phi }}_n\mathit{\theta })+2\underset{j=1,\dots ,J_n}{max}\left|E\left[{\mathrm{\Phi }}_n({\mathit{\theta }}^o-\mathit{\theta })\frac{{\phi }_j}{{\sigma }_j}\right]\right|\left(\sum _{j=1}^{J_n}{\sigma }_j\mid {\widehat{\theta }}_{n,j}-{\theta }_j\mid \right)+E{[{\mathrm{\Phi }}_n({\widehat{\mathit{\theta }}}_n-\mathit{\theta })]}^2\\ \le L({\mathrm{\Phi }}_n\mathit{\theta })+2\gamma {\lambda }_n\left(\sum _{j=1}^{J_n}{\sigma }_j\mid {\widehat{\theta }}_{n,j}-{\theta }_j\mid \right)+E{[{\mathrm{\Phi }}_n({\widehat{\mathit{\theta }}}_n-\mathit{\theta })]}^2,\end{array}$$

where the first equality follows from the fact that E[(R − Φ_nθ^o)φ_j] = 0 for any ${\mathit{\theta }}^o\in [{\mathit{\theta }}_n^{\ast }]$ for j = 1,…, J_n and the last inequality follows from the definition of ${\mathrm{\Theta }}_n^o$.

Based on Lemma A.1 below, we have that on the event Ω₁ ∩ Ω₂(θ) ∩Ω₃(θ),

$$L({\mathrm{\Phi }}_n{\widehat{\mathit{\theta }}}_n)\le L({\mathrm{\Phi }}_n\mathit{\theta })+K_n\frac{\mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid }{{\beta }_n}{\lambda }_n^2.$$

Similarly, when $E[{\mathrm{\Phi }}_2^{(2)}{(X,A)}^T\mid X]=\mathbf{0}$, by Lemma A.2, we have that on the event Ω₁ ∩ Ω₂(θ) ∩ Ω₃(θ),

$$\begin{array}{l}E{({\mathrm{\Phi }}_n^{(2)}{\widehat{\mathit{\theta }}}_n^{(2)}-T_0)}^2\le E{({\mathrm{\Phi }}_n^{(2)}{\mathit{\theta }}^{(2)}-T_0)}^2+2\gamma {\lambda }_n\left(\sum _{j=J_n^{(1)}+1}^{J_n}{\sigma }_j\mid {\widehat{\theta }}_{n,j}-{\theta }_j\mid \right)+E{[{\mathrm{\Phi }}_n^{(2)}({\widehat{\mathit{\theta }}}_n^{(2)}-{\mathit{\theta }}^{(2)})]}^2\\ \le E{({\mathrm{\Phi }}_n^{(2)}{\mathit{\theta }}^{(2)}-T_0)}^2+K_n^{\prime }\frac{\mid M_{\rho {\lambda }_n}^{(2)}(\mathit{\theta })\mid }{{\beta }_n}{\lambda }_n^2.\end{array}$$

The conclusion of the theorem follows from the union probability bounds of the events Ω₁, Ω₂(θ) and Ω₃(θ) provided in Lemmas A.3, A.4 and A.5.

Below we state the lemmas used in the proof of Theorem A.1. The proofs of the lemmas are given in Section S.3 of the supplementary material.

Lemma A.1

Suppose Assumption A.3 holds with ρβ_n ≤ 1. Then for any θ ∈ Θ_n, on the event Ω₁ ∩ Ω₂(θ) ∩ Ω₃(θ), we have

$$\sum _{j=1}^{J_n}{\sigma }_j\mid {\widehat{\theta }}_{n,j}-{\theta }_j\mid \le \frac{20(12\rho {\beta }_n+2\gamma +5)}{(1-2\gamma )(19+2\gamma ){\beta }_n}\mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid {\lambda }_n$$ (A.8)

and

$$E{[{\mathrm{\Phi }}_n({\widehat{\mathit{\theta }}}_n-\mathit{\theta })]}^2\le \frac{130{(12\rho {\beta }_n+2\gamma +5)}^2}{9{(19+2\gamma )}^2{\beta }_n}\mid M_{\rho {\lambda }_n}(\mathit{\theta })\mid {\lambda }_n^2$$ (A.9)

Remark

This lemma implies that θ̂_n is close to each θ ∈ Θ_n on the event Ω₁ ∩ Ω₂(θ) ∩ Ω₃(θ). The intuition is as follows. Since θ̂_n minimizes (4.1), the first order conditions imply that max_j |E_n(R −Φ_nθ̂_n)φ_j/σ̂_j| ≤ λ_n/2. Similar property holds for θ on the event Ω₁ ∩ Ω₃(θ). Assumption A.3 together with event Ω₂(θ) ensures that there is no collinearity in the n × J_n design matrix ${({\mathrm{\Phi }}_n(X_i,A_i))}_{i=1}^n$. These two aspects guarantee the closeness of θ̂_n to θ.

Lemma A.2

Suppose $E[{\mathrm{\Phi }}_n^{(2)}{(X,A)}^T\mid X]=\mathbf{0}$ a.s. and Assumption A.4 holds with ρβ_n ≤ 1. Then for any θ ∈ Θ_n, on the event Ω₁ ∩ Ω₂(θ) ∩ Ω₃(θ), we have

$$\sum _{j=J_n^{(1)}+1}^{J_n}{\sigma }_j\mid {\widehat{\theta }}_{n,j}-{\theta }_j\mid \le \frac{10(12{\beta }_n\rho +2\gamma +5)}{(1-2\gamma )(7-6{\beta }_n\rho ){\beta }_n}\mid M_{\rho {\lambda }_n}^{(2)}(\mathit{\theta })\mid {\lambda }_n$$ (A.10)

and

$$E{[{\mathrm{\Phi }}_n^{(2)}({\widehat{\mathit{\theta }}}_n^{(2)}-{\mathit{\theta }}^{(2)})]}^2\le \frac{20(12\rho {\beta }_n+2\gamma +5)[3(1-2\gamma ){\beta }_n\rho +10(2\gamma +5)]}{9{(2\gamma +19)}^2{\beta }_n}\mid M_{\rho {\lambda }_n}^{(2)}(\mathit{\theta })\mid {\lambda }_n^2$$ (A.11)

Lemma A.3

Suppose Assumption A.2(a) and inequality (A.6) hold. Then $\mathbf{P}({\mathrm{\Omega }}_1^C)\le exp(-k_n^{\prime }n)$, where $k_n^{\prime }=13{(1-2\gamma )}^2/[6(27U_n^2-10\gamma -22)]$.

Lemma A.4

Suppose Assumption A.2(a) holds. Then for any t > 0 and θ ∈ Θ_n, P({Ω₂(θ)}^C) ≤ 2 exp(−t)/3.

Lemma A.5

Suppose Assumptions A.1 and A.2 hold. For any t > 0, if λ_n satisfies condition (A.5), then for any θ ∈ Θ_n, we have P({Ω₃(θ)}^C) ≤ 2 exp(−t)/3.

A.3. Design of simulations in Section 5.1

In this section, we present the detailed simulation design of the examples used in Section 5.1. These examples satisfy all assumptions listed in the Theorems (it is easy to verify that for examples 1–3. Validity of the assumptions for example 4 is addressed in the remark after example 4). In addition, Θ_n defined in (4.4) is non-empty as long as n is sufficiently large (note that the constants involved in Θ_n can be improved and are not that meaningful. We focused on a presentable result instead of finding the best constants).

In examples 1 – 3, X = (X₁,…, X₅) is uniformly distributed on [−1, 1]⁵. The treatment A is then generated independently of X uniformly from {−1, 1}. Given X and A, the response R is generated from a normal distribution with mean Q₀(X, A) = 1+2X₁ +X₂ +0.5X₃ +T₀(X, A) and variance 1. We consider the following three examples for T₀.

1. T₀(X, A) = 0 (i.e. there is no treatment effect).

2. T₀(X, A) = 0.424(1 − X₁ − X₂)A.

3. T₀(X, A) = 0.446sign(X₁)(1 − X₁)²A.

Note that in each example T₀(X, A) is equal to the treatment effect term, Q₀(X, A) − E[Q₀(X, A)|X]. We approximate Q₀ by = {(1, X, A, XA)θ: θ ∈ ℝ¹²}. Thus in examples 1 and 2 the treatment effect term T₀ is correctly modeled, while in example 3 the treatment effect term T₀ is misspecified.

The parameters in examples 2 and 3 are chosen to reflect a medium effect size according to Cohen’s d index. When there are two treatments, the Cohen’s d effect size index is defined as the standardized difference in mean responses between two treatment groups, i.e.

$$es=\frac{E(R\mid A=1)-E(R\mid A=-1)}{{([\mathit{Var}(R\mid A=1)+\mathit{Var}(R\mid A=-1)]/2)}^{1/2}}.$$

Cohen [7] tentatively defined the effect size as “small” if the Cohen’s d index is 0.2, “medium” if the index is 0.5 and “large” if the index is 0.8.

In example 4, X is uniformly distributed on [0, 1]. Treatment A is generated independently of X uniformly from {−1, 1}. The response R is generated from a normal distribution with mean Q₀(X, A) and variance 1, where $Q_0(X,1)={\sum }_{j=1}^8{\vartheta }_{(1),j}{1}_{X<u_{(1),j}},Q_0(X,-1)={\sum }_{j=1}^8{\vartheta }_{(-1),j}{1}_{X<u_{(-1),j}}$, and ϑ’s and u’s are parameters specified in (A.12). The effect size is small.

$$\begin{array}{l}({\vartheta }_{(1),1},\dots ,{\vartheta }_{(1),8})=(-0.781,0.730,0.635,0.512,-2.278,1.347,1.155,-0.030);\\ ({\vartheta }_{(-1),1},\dots ,{\vartheta }_{(-1),8})=(-2.068,1.520,-0.072,-0.637,1.003,-0.611,-0.305,1.016);\\ (u_{(1),1},\dots ,u_{(1),8})=(0.028,0.144,0.171,0.298,0.421,0.443,0.463,0.758);\\ (u_{(-1),1},\dots ,u_{(-1),8})=(0.061,0.215,0.492,0.544,0.6302,0.650,0.785,0.909).\end{array}$$ (A.12)

We approximate Q₀ by Haar wavelets

$$\mathcal{Q}=\left\{{\theta }_{(0),0}{h}_0(X)+\sum _{lk}{\theta }_{(0),lk}{h}_{lk}(X)+\left({\theta }_{(0),1}{h}_0(X)+\sum _{lk}{\theta }_{(1),lk}{h}_{lk}(X)\right)A:\theta .,.\in \mathbb{R}\right\},$$

where h₀(x) = 1_x∈[0,1] and h_lk(x) = 2^l/2 (1_{2^lx∈[k+1/2,k+1)} − 1_{2^lx∈[k,k+1/2)}) for l = 0,…, l̄_n. We choose l̄_n = ⌊3log₂ n/4⌋ − 2. For a given l and sample ${(X_i,A_i,R_i)}_{i=1}^n$, k takes integer values from ⌊2^l min_i X_i⌋ to ⌈2^l max_i X_i⌉ − 1. Then J_n = 2^{⌊3 log₂n/4⌋} ≤ n^3/4.

Remark

In example 4, we allow the number of basis functions J_n to increase with n. The corresponding theoretical result can be obtained by combining Theorem 3.1 and Theorem A.1. Below we demonstrate the validation of the assumptions used in the theorems.

Theorem 3.1 requires that the randomization probability p(a|x) ≥ S⁻¹ for a positive constant for all (x, a) pairs and the margin condition (3.3) or (3.6) holds. According the generative model, we have that p(a|x) = 1/2 and condition (3.6) holds.

Theorem A.1 requires Assumptions A.1 - Assumptions A.4 hold and Θ_n defined in (A.4) is non-empty. Since we consider normal error terms, Assumption A.1 holds. Note that the basis functions used in Haar wavelet are orthogonal. It is also easy to verify that Assumptions A.3 and A.4 hold with β_n = 1 and Assumption A.2 holds with U_n = n^3/8/2 and ${\eta }_{1,n}\le \mathit{constant}+\mathit{constant}\times {\left|\right|{\mathit{\theta }}_n^{\ast }\left|\right|}_0$ (since each $\mid {\phi }_j{\mathit{\theta }}_{n,j}^{\ast }\mid =\mid {\phi }_{j}E({\phi }_{j}R)\mid \le \mathit{constant}\times \mid {\phi }_j\mid E\mid {\phi }_j\mid \le O(1)$). Since Q₀ is piece-wise constant, we can also verify that ${\left|\right|{\mathit{\theta }}_n^{\ast }\left|\right|}_0\le O(logn)$. Thus for sufficiently large n, Θ_n is non-empty and (A.6) holds. The RHS of (A.5) converges to zero as n → ∞.