On Sparse representation for Optimal Individualized Treatment Selection with Penalized Outcome Weighted Learning

Abstract

As a new strategy for treatment which takes individual heterogeneity into consideration, personalized medicine is of growing interest. Discovering individualized treatment rules (ITRs) for patients who have heterogeneous responses to treatment is one of the important areas in developing personalized medicine. As more and more information per individual is being collected in clinical studies and not all of the information is relevant for treatment discovery, variable selection becomes increasingly important in discovering individualized treatment rules. In this article, we develop a variable selection method based on penalized outcome weighted learning through which an optimal treatment rule is considered as a classification problem where each subject is weighted proportional to his or her clinical outcome. We show that the resulting estimator of the treatment rule is consistent and establish variable selection consistency and the asymptotic distribution of the estimators. The performance of the proposed approach is demonstrated via simulation studies and an analysis of chronic depression data.

1. Introduction

It is well known that patients possess strong heterogeneity in response to treatments in modern clinical studies, especially in chronic diseases. Consequently, a drug may have significant treatment effect for some patients with certain characteristics, but not work for other patients. Treating each individual based on his or her genomic or prognostic information rather than via “one size fits all” approach can thus potentially improve drastically the effect of treatment for public health. This significantly motivates interest in discovering personalized treatment strategies in statistical research, where an optimal personalized treatment strategy is a set of treatment rules tailored to individuals, to maximize long-term clinical outcomes for individual patients, see Murphy (2003); Robins (2004); Moodie et al. (2007).

Various methods have been proposed in the statistical literature to estimate the optimal personalized treatment strategy. Q-learning (Watkins, 1989; Watkins & Dayan, 1992; Murphy, 2005; Zhao et al., 2011; Chakraborty et al., 2010; Song et al., 2014) and A-learning (Murphy, 2003; Robins, 2004) are two backward induction methods for deriving optimal multistage treatment regimes. Other related approaches include likelihood-based methods (Thall et al., 2000, 2002, 2007) and semiparametric methods (Lunceford et al., 2002; Wahed & Tsiatis, 2004, 2006; Moodie et al., 2009; Zhang et al., 2012). Zhao et al. (2012) proposed outcome weighted learning, which can be viewed as a weighted classification problem using the covariate information weighted by the individual response to achieve the goal of maximizing the overall outcome. Since more and more individual-level information are collected in clinical studies, especially in multi-stage clinical trials, these approaches are often challenged by the curse of dimensionality in the presence of high-dimensional covariates. Variable selection hence becomes important in discovering individualized treatment rules.

In spite of a large effort being devoted to variable selection methods, the topic of variable selection for personalized and dynamic treatment regimens has received little attention. Some variable selection techniques used in this area include the Lasso (Loth et al., 2007), decision trees (Ernst et al., 2005) and Bayesian variable selection (Chen et al., 2009). Biernot & Moodie (2010) conducted numerical studies to compare several variable selection methods. In some clinical trials where medical decision making is needed, qualitative interaction tests (Gail & Simon, 1985; Piantadosi & Gail, 1993; Yan, 2004) have been used to test a small number of expert determined pre-specified interactions, such as Allhat et al. (2002); Reynolds et al. (2006). Many of the tests were designed for testing only qualitative interactions between categorical variables and the treatment. Moreover, when the number of covariates is large, these tests are too conservative when controlling the error rate due to multiple testing.

When the dimension of the covariate space is high and there are many irrelevant variables for decision making, imposing sparsity in the parameters via an automatic variable selection procedure can significantly enhance the decision making accuracy. Penalized methods have also been studied to identify variables important for making treatment decisions. Qian & Murphy (2011) developed a two-stage procedure in the framework of Q-learning, where they used L₁ penalized least squares to estimate optimal treatment regimes. Lu et al. (2013) proposed a penalized quadratic loss in the framework of A-learning. Gunter et al. (2011) proposed variable selection methods for qualitative interactions, where two variable-ranking quantities were presented. Existing penalization methods are primarily designed for linear regression models, which may not be appropriate for variable selection in treatment decision making when the true model is not correctly specified. We are therefore motivated to develop variable selection methods that do not depend on the parametric modeling of the value function.

In this paper, we propose to adopt the outcome weighted learning framework to simultaneously estimate the optimal decision rule and incorporate sparsity in a way that retains the computational advantages and theoretical validity. Our proposed method is called penalized outcome weighted learning (POWL). We demonstrate later in the paper that POWL can achieve comparable or better classification accuracy while, at the same time, choosing relevant features. The main idea is to introduce the SCAD penalty proposed in Fan & Li (2001) to outcome weighted learning. The resulting optimization problem is a linear program and can be efficiently solved. Our approach can handle very high-dimensional state spaces. Furthermore, we study Fisher consistency of the linear POWL. We also investigate asymptotic properties of the variable coefficients in the SVM solution for linear classification. We focus on the case where the decision function is a linear function of the covariates. Various variable selection methods were studied in recent years to encourage sparsity in support vector machines (Cortes & Vapnik, 1995). References include Zhu et al. (2003); Wu et al. (2008); Park et al. (2012) etc. Our work can be viewed as a penalized weighted SVM.

The rest of the paper is organized as follows. The proposed POWL is introduced in Section 2. In Section 3, we study the theoretical properties of POWL and numerical studies are presented in Section 4.We apply the proposed method to a chronic depression data set in Section 5, and conclude with a brief discussion in Section 6.

2. Method

2.1. Outcome-Weighted Learning (OWL)

Consider data from a randomized trial. By notation, we use A to denote treatment assignment, which takes values −1 and 1. The probability of treatment assignment A is given as P(A = 1) = π and P(A = −1) = 1 − π, where 0 < π < 1. We use X = (X₁,…, X_p)′ to denote p-dimensional biomarker and prognostic information associated with the patient and Let X̃ = (1, X′)′. Let R be the clinical outcome of interest (assuming large values are desirable) which we also call the “reward.” Let f (x, r) and g(x, r) denote the conditional density of (X, R) given A = 1 and −1 respectively. In this randomized trial setting, A is randomly assigned to patients and the observed data from n i.i.d patients has the form (A_i, X_i, R_i), i = 1,…, n. Our goal is to identify a deterministic decision rule, d(x), which takes a given value x of X and returns a treatment choice from a space 𝒜. We denote the distribution of (X, A, R) by P and expectation with respect to this distribution by E. Let P^d denote the distribution of (X,A, R) when A = d(X), the treatment is determined by the rule d. The expectation with respect to this distribution is denoted by E^d, where the individualized treatment rule (ITR) d(x) is used to assign treatments. Define the value function as V (d) = E^d (R). Thus, an optimal individualized treatment rule, d₀, is a rule that has the maximal value, i.e., d₀ is the maximizer of V(d) over decision rules d. Specifically, V(d₀) = E{max_A∈𝒜 Q(X, d₀(X))} where Q(X, A) ≡ E[R|A = d(x), X = x]. The expected reward under ITR 𝒟 is given as

$$E^{𝒟}(R)=\int Rd{P}^{𝒟}=\int R\frac{d{P}^{𝒟}}{dP}dP=E\left[\frac{I(A=𝒟(\mathbf{X}))}{A\pi +(1-A)/2}R\right].$$ (1)

It can be seen from (1) that maximizing V (𝒟) is equivalent to a weighted classification problem, where we classify subjects with X according to treatment A but weight each subject by R/(A_π + (1 − A)/2). Based on this idea, outcome weighted learning was developed in Zhao et al. (2012). Because direct optimization of (1) is intractable due to the nonconvex and discontinuous 0−1 loss, Zhao et al. (2012) propose to use convex relaxation in combination with techniques applied in support vector machines (Cortes & Vapnik, 1995).

In the next section, we generalize this learning approach to allow simultaneous maximization of the value function and variable selection, which we call penalized outcome-weighted learning (POWL). We consider the situation where the optimal decision rule belongs to a linear function class of the feature covariates.

2.2. Penalized Outcome-weighted Learning (POWL)

Let p_λ(β) denote some penalty function, where β ∈ ℝ^p+1 is the parameter of interest and λ is the tuning parameter. The POWL minimizes the following objective function

$$Q_n(\mathit{\beta })=\frac{1}{n}\sum _{i=1}^n\frac{R_i}{A_i\pi +(1-A_i)/2}{(1-A_{i}h({\mathbf{X}}_i,\mathit{\beta }))}_{+}+\sum _{j=1}^p{p}_{\lambda }(|{\beta }_j|),$$ (2)

where h(·) is the decision function, $\mathit{\beta }=({\beta }_0,{\mathit{\beta }}_{-}^{\prime })\prime$ and β₋ = (β₁,…, β_p)′, (z)₊ = max(z, 0). Choices of the penalty functions include the lasso (Tibshirani, 1996) and the SCAD (Fan and Li, 2001). The limit of Q_n(β) is defined as

$$Q(\mathit{\beta })=E\left[\frac{R}{A\pi +(1-A)/2}{(1-Ah(\mathbf{X};\mathit{\beta }))}_{+}\right].$$ (3)

We note that it is the same as the population objective function of the OWL proposed in Zhao et al. (2012).

Suppose that the decision function h(X; β) minimizing (3) is a linear function of X with true parameter value β = β⁰, that is, h(X, β) = X̃′β⁰ and β⁰ = argmin_βQ(β). Then the corresponding individualized optimal rule will assign a subject with prognostic value X into treatment 1 if X̃′β⁰ > 0 and −1 otherwise. Let β̂ = argmin_βQ_n(β). The POWL will classify a subject (X_i,A_i, R_i) by the sign of h(X_i, β̂) correspondingly. Define $ℳ=\{j:{\beta }_j^0\ne 0\}$. Let s = |ℳ|, the cardinality of the non-sparse set ℳ. We aim to estimate β⁰ and recover the set ℳ. To facilitate the theoretical derivation and the numerical calculation, we consider the SCAD penalty proposed in Fan & Li (2001) defined as

$$p_{{\lambda }_n}^{\prime }(\theta )={\lambda }_n\{1(\theta \le {\lambda }_n)+\frac{{(a{\lambda }_n-\theta )}_{+}}{{\lambda }_n(a-1)}1(\theta >{\lambda }_n)\},$$

where a > 2 and λ_n > 0 are the tuning parameters.

We propose to use the one-step estimator in Zou & Li (2008) to approximate the penalty term. Minimizing (2) can be rewritten as

$$\underset{\mathit{\beta },\Vert {\mathit{\beta }}_{-}\Vert =1}{\text{max}}\sum _{i=1}^n\frac{R_i}{A_i\pi +(1-A_i)/2}{\xi }_i+\sum _{i=0}^p{p}_{{\lambda }_n}^{\prime }({\beta }_i)|{\beta }_i|,$$

$$\text{subject to}|A_i({\beta }_0+{\mathit{\beta }}_{-}^{\prime }{\mathbf{X}}_i)|\ge (1-{\xi }_i),{\xi }_i\ge 0,$$

where ξ_i is a slacking variable for subject i to allow a small portion of wrong classification. Therefore we use linear programming to solve the above problem.

3. Theoretical Results

In this section, we study the theoretical properties of the proposed POWL with SCAD penalty. First we consider the Fisher consistency of the linear POWL. It was shown in Zhao et al. (2012) that the decision function based on the surrogate loss Q(β) possess Fisher consistency. Since the population version of our objective function remains the same as OWL, our procedure also possess Fisher consistency, that is, the population version of our estimation procedure is the same as the target of the estimation, and the population minimizer of the surrogate objective function yields a classification rule with the same sign as that of the Bayes rule.

We define the score as:

$$S(\mathit{\beta })=-E[\frac{R}{A\pi +(1-A)/2}1(1-Ah(\mathbf{X},\mathit{\beta })\ge 0)A\tilde{\mathbf{X}}].$$

We also define

$$H(\mathit{\beta })=E[\frac{R}{A\pi +(1-A)/2}\delta (1-Ah(\mathbf{X},\mathit{\beta }))A\tilde{\mathbf{X}}\tilde{\mathbf{X}}\prime ],\text{and}$$

$$G(\mathit{\beta })=E[\frac{R}{A\pi +(1-A)/2}1(1-Ah(\mathbf{X},\mathit{\beta })\ge 0)A\tilde{\mathbf{X}}\tilde{\mathbf{X}}\prime ],$$

where the δ function is the Dirac delta function, that is, on the real number line it is zero everywhere except at zero, with the integral function as the indicator function 1(t ≥ 0).

Next we establish consistency and convergence rates of the estimator β̂ under the following conditions. We use lower case to represent the realizations of the random variables. Denote f (x) and g(x) as the conditional densities of x given A = 1 and −1 respectively.

• A1The densities f(x, r) and g(x, r) are continuous with finite second moments.
• A2There exists B(x₀, δ₀), a ball centered at x₀ with radius δ₀ > 0 such that f(x) > C₁, g(x) > C₁ for any x ∈ B(x₀, δ₀).
• A3For some 1 ≤ j ≤ p, E(X_jR|A = 1) ≠ E(X_jR|A = −1).
• A4
  Let M⁺ = {x ∈ 𝒳|x′β⁰ = 1} and M⁻ = {x ∈ 𝒳|x′β⁰ = −1}. For an orthogonal transformation ϕ_j that maps ${\mathit{\beta }}_{+}^0/\Vert {\mathit{\beta }}_{+}^0\Vert$ to the unit vector e_j whose j−th element is one and the other elements are zero, for some 1 ≤ j ≤ d, there exist rectangles

  $$D^{+}=\{\mathbf{x}\in M^{+}:l_i\le {({\mathrm{\varphi }}_j\mathbf{x})}_i\le v_i,\text{for}i\ne j\},\text{and}$$

  $$D^{-}=\{\mathbf{x}\in M^{-}:l_i\le {({\mathrm{\varphi }}_j\mathbf{x})}_i\le v_i,\text{for}i\ne j\}.$$

Condition A1 and A4 are needed for the Hessian matrix H(β) to be continuous and positive definite near β⁰. Condition A2 and A3 ensure the weighted classification problem is nonseparable and β⁰ is nonzero.

Theorem 1

Under Conditions A1–A4, if λ_n → 0 and $\sqrt{n}{\lambda }_n\to \infty$, then with probability tending to one we have

$$\Vert {\hat{\mathit{\beta }}}_n-{\mathit{\beta }}^0\Vert =O_p(n^{-1/2}).$$

To discuss the oracle properties of the POWL with SCAD penalty, we need the following notation. Let ${\mathit{\beta }}^0=({\mathit{\beta }}_1^{0\prime },0\prime )\prime$, where ${\mathit{\beta }}_1^0\in {ℝ}^{S+1}$ represents the true value for β₁, the nonzero elements of β. Let

$$H(\mathit{\beta })=\left(\begin{array}{cc}{H}_{11}(\mathit{\beta })\hfill & H_{12}(\mathit{\beta })\hfill \\ H_{21}(\mathit{\beta })\hfill & H_{22}(\mathit{\beta })\hfill \end{array}\right),\text{and}G(\mathit{\beta })=\left(\begin{array}{cc}{G}_{11}(\mathit{\beta })\hfill & G_{12}(\mathit{\beta })\hfill \\ G_{21}(\mathit{\beta })\hfill & G_{22}(\mathit{\beta })\hfill \end{array}\right),$$

where H₁₁(β) and G₁₁(β) are the top (1 + s) square sub-matrix of H(β) and G(β), respectively. The sparsity and the oracle property of the estimators are established in the following theorem:

Theorem 2

Denote $\hat{\mathit{\beta }}=({\hat{\mathit{\beta }}}_1^{\prime },{\hat{\mathit{\beta }}}_2^{\prime })\prime$, where β̂₁ ∈ ℝ^s+1 estimates β₁ and β̂₂ ∈ ℝ^p−s estimates β₂. Under Conditions A1–A4, if λ_n → 0 and $\sqrt{n}{\lambda }_n\to \infty$, then lim_n→∞ P(β̂₂ = 0) = 1 and $\sqrt{n}({\hat{\mathit{\beta }}}_1-{\mathit{\beta }}_1^0){\to }_{d}N(0,H_{11}^{-1}({\mathit{\beta }}^0)G_{11}({\mathit{\beta }}^0)H_{11}^{-1}({\mathit{\beta }}^0))$.

4. Simulation Studies

Zhao et al. (2012) compared the numerical performance of OWL and the L₁ method in Qian & Murphy (2011), where OWL demonstrated better performance in value function estimation. Therefore in this section we focus on comparing the numerical performance of OWL and POWL in variable selection and value function estimation. We generated 50 dimensional vectors of prognostic variables X₁,…, X₅₀. They are independently uniformly distributed random variables on [−1, 1]. The treatment A takes values −1 and 1 and is independent of the covariates. The response variable R is normally distributed with mean Q₀ = 1 + 2X₁ + X₂ + 0.5X₃ + T(X, A) and variance 1. The term T(X, A) represents the interaction between treatment and prognostic variables. The first s variables are relevant while the remaining p − s variables are noise variables. We consider the following two scenarios with different degrees of sparsity level:

1. Scenario 1: T(X, A) = (X₁ + X₂)A;

2. Scenario 2: T(X, A) = (X₁ + X₂ + X₃ − X₄ + X₅ − X₆)A.

Therefore the optimal rule for Scenario 1 is I(X₁ + X₂ > 0) and that for Scenario 2 is I(X₁ + X₂ + X₃ − X₄ + X₅ − X₆ > 0). Based on large scale Monte Carlo simulations, the optimal values for two scenarios are 1.66 and 2.13 respectively. We generated n training samples with n ranging over 30, 100, 200, 400 and 800. For each scenario we estimate the optimal ITR by applying OWL and POWL and repeated the procedure on 200 independent data sets. The BIC criteria is used to select the optimal values of the tuning parameters. The grid for the tuning parameter λ_n is {2⁻⁵, 2⁻⁴,…, 2⁵}.

The performances of the methods are evaluated by the following four criteria. The first is to evaluate the value function using the estimated optimal individual treatment rule when applying to an independent validation data set with sample size 10000. We evaluate the estimated value function for any ITR with the method in Murphy et al. (2001). The second is to evaluate the misclassification rates of the estimated optimal ITR from the true optimal ITR using the validation data. The third is to evaluate the size of the selected model. The fourth is to evaluate the number of true positives in the selected model. Because the size of the selected model and the number of true positives are closely related with the choice of tuning parameter, we also record the minimal model sizes that are needed to cover the true model to gauge the difficulty of the problem. The simulation results are summarized in Tables 1 and 2.

Table 1.

Simulation results based on 200 independent datasets for Scenario 1. The sample mean of both value functions and misclassification rates (“MR”) with the associated sample variance in parenthesis are presented. The sample median of the minimum model size, the size of the selected model and the number of true positives are also recorded with the associated robust standard deviation in parenthesis. “MR” denotes misclassification rate. “MMMS” denotes minimum model size. “NV” denotes number of selected variables. “TP” denote number of true positives. The optimal value is 1.66.

Sample size	Methods	MR	Value	MMMS	NV	TP
30	OWL	0.3686(0.0312)	0.9008(0.1849)
	POWL	0.3682(0.0349)	0.9015(0.2011)	27(37.52)	5(5.36)	0(1.34)
100	OWL	0.3352(0.0017)	1.1308 (0.0072)
	POWL	0.3049(0.0130)	1.3765(0.0460)	20(44.22)	6(8.04)	1(1.34)
200	OWL	0.2912(0.0225)	1.0113(0.2650)
	POWL	0.1722(0.0135)	1.2153(0.3874)	7(6.70)	3(2.68)	1(1.34)
400	OWL	0.2580(0.0074)	1.2875(0.1684)
	POWL	0.1842(0.0065)	1.4152(0.1983)	2(0)	2(1.34)	2(1.34)
800	OWL	0.2045(0.0004)	1.5209(0.0011)
	POWL	0.0649(0.0013)	1.6474(0.0008)	2(0)	4(2.68)	2(0)

Table 2.

Simulation results based on 200 independent datasets for Scenario 2. The sample mean of both value functions and misclassification rates (“MR”) with the associated sample variance in parenthesis are presented. The sample median of the minimum model size, the size of the selected model and the number of true positives are also recorded with the associated robust standard deviation in parenthesis. The notations are the same as these in Table 1. The optimal value is 2.13.

Sample size	Methods	MR	Value	MMMS	NV	TP
30	OWL	0.4050(0.0018)	1.3286(0.0218)
	POWL	0.4280(0.0036)	1.2517(0.0448)	42(14.74)	6(2.68)	0(1.34)
100	OWL	0.3579(0.0020)	1.4801(0.0208)
	POWL	0.2959(0.0035)	1.6761(0.0304)	34(20.10)	10(5.36)	4(2.68)
200	OWL	0.2749(0.0010)	1.7357(0.0092)
	POWL	0.2548(0.0025)	1.7824(0.0121)	29(40.20)	8(4.02)	5(1.34)
400	OWL	0.2912(0.0225)	1.0113(0.2650)
	POWL	0.1722(0.0135)	1.2153(0.3874)	10(0)	11(5.36)	6(0)
800	OWL	0.1345(0.0002)	2.0333(0.0011)
	POWL	0.0523(0.0009)	2.1195(0.0014)	7(0)	6(2.68)	6(0)

We report the sample mean of both value functions and misclassification rates with the associated sample variance in parenthesis. The sample median of the minimum model size, the size of the selected model and the number of true positives are also recorded with the associated robust standard deviation in parenthesis. As the sample size increases, the performance of OWL and POWL increase in the sense that the value function estimates get closer and closer to the optimal value function and the misclassification rates continue decreasing. POWL provides bigger value function estimates and smaller misclassification rates in all simulation settings. In terms of the variable selection results, POWL can recover the true model better and better as the sample size grows. The number of true positives increases as the sample size increases.

5. Data Analysis

We apply the proposed method to the data from the Nefazodone-CBASP trial (Keller et al., 2000). This trial was conducted to compare the efficacy of several alternative treatments for patients with chronic depression. In the study, 681 patients with nonpsychotic chronic major depressive disorder (MDD) were randomized to either Nefazodone, cognitive behavioral-analysis system of psychotherapy (CBASP), or to a combination of both treatments. The primary outcome was the score on the 24-item Hamilton Rating Scale for Depression (HRSD). Low HRSD scores are desirable.

We used a subset of the Nefazodone-CBASP data consisting of 647 patients, where 211, 216 and 220 patients were assigned to the combined treatment, Nafazodone and CBASP group, respectively. Among the three treatments, pairwise comparisons showed that the combination treatment significantly lowered the HRSD scores compared to either of the single treatments. There was no overall difference between the two single treatments.

We used the proposed POWL to estimate the optimal individualized treatment rule with 50 pretreatment variables. Pairwise comparisons were conducted among the three treatments. We calculated the value functions from a five-fold cross-validation analysis, where the estimated rules were obtained based on four folds of the data and the estimated value functions were obtained use the remaining fifth fold of the data. The value functions calculated this way should better represent expected value functions for future subjects, as compared to calculating value functions based on the training data. The averages of the cross-validation value functions from the POWL and the OWL approach are presented in Table 3.

Table 3.

Mean depression scores from cross-validation procedure with OWL and POWL. "NVs" denotes the number of variables selected via POWL.

	OWL	POWL
	HRSD	HRSD	NVs
Nefazodone vs CBASP	13.86	13.67	29
Combination vs Nefazodone	13.53	13.11	33
Combination vs CBASP	15.63	15.22	35

As can be seen in Table 3, the POWL approach produced slightly larger value functions, corresponding to smaller HRSD values compared with the OWL approach. When comparing combination treatment with nefazodone only, both OWL and POWL recommended the combination treatment to all patients in the validation data in each round of the cross validation procedure. When the two single treatments are studied, there are only negligible differences in the estimated value functions for the two methods and the selection results also indicate an insignificant difference between them. Meanwhile, POWL needs less variables for decision making. In summery, POWL not only yields individualized treatment rules with the best clinical outcomes, but also enjoys more parsimonious decision rules.

6. Discussion

In this article, we proposed a new variable selection method for optimal treatment decision making, which does not need modeling of the value function. Our method can be applied to data with continuous outcomes and binary treatment options from a clinical trial or an observational study. Simulation studies demonstrate that our method can identify the most important variables under various settings and can provide a good estimated optimal treatment regime which has a small error rate and a high average outcome value.

Dynamic treatment regimes, which are sequential decision rules for individual patients that can adapt over time, are more useful than single-stage decision rules in some applications, see the discussion in Murphy (2003). There are great recent interests on developing statistical inference for dynamic treatment regimes (Robins, 2004; Moodie et al., 2007; Zhao et al., 2011; Song et al., 2014). It will be interesting and useful to generalize the proposal approach for variable selection in developing dynamic treatment regimes.