SUMMARY
It is well accepted that individualized treatment regimes may improve the clinical outcomes of interest. However, positive treatment effects are often accompanied by certain side effects. Therefore, when choosing the optimal treatment regime for a patient, we need to consider both efficacy and safety issues. In this article, we propose to model time to a primary event of interest and time to severe side effects of treatment by a competing risks model and define a restricted optimal treatment regime based on cumulative incidence functions. The estimation approach is derived using a penalized value search method and investigated through extensive simulations. The proposed method is applied to an HIV dataset obtained from Health Sciences South Carolina, where we minimize the risk of treatment or virologic failures while controlling the risk of serious drug-induced side effects.
Keywords: Competing risks data, Cumulative incidence function, Optimal treatment regime, Side effects, Value search method
1. Introduction
Generally, there is no uniform best treatment for all patients because of individual heterogeneity. Personalized medicine is a paradigm that aims to tailor treatment to maximize its effect according to the patient’s characteristics. The treatment effect may be determined by how well the treatment can improve the clinical outcomes of interest. For example, angiotensin converting enzyme inhibitors are evaluated according to how well they can control blood pressure in hypertension studies, while in HIV studies, different antiretroviral agents are compared with respect to their ability to control the HIV viral load and CD4 counts. A number of studies have been conducted and published that address this question, including Q-learning (Watkins and Dayan, 1992), A-learning (Murphy, 2003, 2005), direct value search methods (Zhang and others, 2012, 2013), and outcome-weighted learning (Zhao and others, 2012, 2015a). The treatment can also target survival time such as how long the HIV treatment can suppress a viral load under 200 copies/mL. When the primary endpoint to evaluate the treatment effect is survival time, Zhao and others (2015b) and Bai and others (2017) proposed doubly robust estimators of the optimal treatment regime from a classification perspective. Jiang and others (2017a) proposed an optimal treatment regime estimation method that maximizes 
-year survival probability; Jiang and others (2017b) extended it to maximize a user-specified function of the survival curve; and Yavuz and others (2018) proposed nonparametric estimators for the cumulative incidence function (CIF) of dynamic treatment regimes, where treatment decisions need to be tailored over time according to the patient’s responses to previous treatments.
Our study is motivated by the HIV dataset obtained from Health Sciences South Carolina (HSSC). HSSC (https://www.healthsciencessc.org/) is a biomedical research collaborative consisting of six of the state’s largest health systems, namely the Medical University of SC, Spartanburg Regional Healthcare System, McLeod Health, AnMed Health, Prisma Health, and Self Regional Healthcare. The three most commonly used antiretroviral treatment (ART) classes are nucleoside reverse transcriptase inhibitors (NRTIs), non-nucleoside reverse transcriptase inhibitors (NNRTIs), and protease inhibitors (PIs) (Günthard and others, 2016). Drugs in the same class share common properties, whereas drugs in a different class have different treatment effects. NNRTIs are associated with faster virologic suppression, and PIs recover more CD4 cells (World Health Organization, 2016; Günthard and others, 2016). In general, modern ART consists of a combination of at least three agents from two classes (Günthard and others, 2016). Common combinations such as “NNRTIs+NRTIs” and “PIs+NRTIs” have also been compared in the literature regarding their treatment effects measured by the level of virologic suppression or CD4 recovery. (Staszewski and others, 1999; Haubrich and others, 2009; Smith and others, 2009; Borges and others, 2016). Jiang and others (2017b) considered the optimal regime of “NNRTIs+NRTIs” and “PIs+NRTIs” to maximize the longest initial treatment duration based on a dataset from an HIV/AIDS clinical observational study.
However, it is worthwhile to point out that all ART drugs cause side effects. Some side effects such as headaches or occasional dizziness may not be serious. Others such as swelling of the throat and tongue, damage to the liver, and myocardial infarction can be life-threatening (Dybul and others, 2002; Worm and others, 2010). Serious, life-threatening events (e.g., abacavir hypersensitivity reaction, symptomatic hepatotoxicity, or severe cutaneous reactions) require the immediate discontinuation of all ART drugs and re-initiation of an alternative regimen without overlapping toxicity (Health and Human Services, 2017). Recently, Wang and others (2018) considered both efficacy and safety issues in deriving an optimal treatment regime via maximizing clinical benefit under a constraint on the average risk.
Let us define “risk 1” as treatment or virologic failure, which is monitored by either CD4 counts (
500 cells/
) or HIV viral load (
200 copies/mL), and “risk 2” as the drug-induced, long-term side effects, including kidney problems, liver damage, and nervous system/psychiatric effects such as insomnia, dizziness, depression, and suicidal thoughts (Simpson and others, 2014). In the HSSC dataset, a total of 229 patients took drug combinations “NNRTIs+NRTIs” or “PIs+NRTIs” with complete laboratory measures. We are interested in days to the first onset of either risk after the first drug initiation on record. Thus, days to either risk, whichever came first after drug administration, were recorded. Figure 1 illustrates the CIFs of the two HIV treatments separately for black and non-black patients. In Figure 1, we observed that patients taking “NNRTIs+NRTIs” generally had a lower risk of treatment or virologic failure, but a higher risk of having serious side effects than those taking “PIs+NRTIs” among non-black patients. Moreover, the patterns of risks with these two types of drugs were reversed among black patients. Therefore, there are no consistent criteria by which to assign “NNRTIs+NRTIs” or “PIs+NRTIs” to each individual. Based on HIV patients’ characteristics in the HSSC data, whether we can achieve an optimal treatment regime that can minimize the risk of treatment or virologic failure while controlling the risk of long-term side effects under a tolerable limit is a challenging problem in practice. Such an optimal treatment regime may provide useful guidance for practitioners for antiretroviral drug consultation.
Fig. 1.
CIFs for HIV data
To achieve an optimal treatment regime that can balance treatment efficacy and side effects are the main purpose of this study. For simplicity, we consider only two risks here. The primary risk of interest is treatment failure and the other is the risk of adverse drug effects. First onset time of either risk is recorded and treated as the competing risk data. Estimation methods for competing risks data have been studied extensively as evidenced by the literature (Gray, 1988; Fine and Gray, 1999; Klein and Andersen, 2005; Sun and others, 2006; Lu and Peng, 2008; Mao and Lin, 2017). However, few methods exist for studying the optimal treatment regime under the competing risks framework. Therefore, there is an emerging need to develop an optimal individualized treatment regime that achieves balance between risk 1 and risk 2 in practice. Toward this goal, we define a restricted optimal treatment regime that minimizes the 
-year CIF of the main risk while controlling the 
-year cumulative incidence of the other risk under a predetermined level and deriving its estimation procedure via a penalized value search method.
The rest of the article is organized as follows: in Section 2, the notation and definition of the restricted optimal treatment regime for competing risks data are introduced. An estimator for the proposed restricted optimal treatment regime is presented in Section 3, along with the details of the implementation procedure. Simulation studies are conducted to examine the empirical properties of the proposed estimator in Section 4. An application to the HSSC HIV data is presented in Section 5. Finally, some conclusions and discussions are summarized in Section 6.
2. Restricted optimal treatment regime
Let 
and 
denote the event times of risks 1 and 2, respectively, and 
denote the right-censoring time. Define 
, 
, 
and 
as an indicator for competing risks. In addition, let 
denote the 
-dimensional vector of baseline covariates related to the prediction models for the event times, and 
denote the 
-dimensional vector of baseline covariates related to the propensity score model for treatment assignment. Note that 
and 
may share some common variables. We assume 
is the actual treatment received and two treatment options, 
, are available. A treatment regime 
is a mapping from 
to 
, where 
is the support for 
. For example, under the linear decision rule, 
, where 
. The observed data include 
.
To define the restricted optimal treatment regime, let 
denote the potential event times for risk 
, 
, if treatment 
was assigned to the patient, where 
. Moreover, let 
denote the potential censoring time. Define 
, 
and 
. In addition, let 
denote the survival function of 
. Then the CIF of 
is given by
 |
(2.1) |
where 
is the cause-specific hazard function for 
, i.e. 
.
Similarly, define 
, which denotes the potential event times for risk 
, 
, if all patients were treated according to the regime 
. Let 
. We can define 
for the overall survival function given the regime 
and define the cause-specific hazard function 
for 
given the regime 
accordingly. Then, we can obtain 
by replacing 
with 
in the definition of 
.
Our proposed restricted optimal treatment regime is defined by
 |
where 
is a fixed time point of interest, and 
. In other words, the proposed restricted optimal treatment regime minimizes the 
-year cumulative incidence for risk 1 while controlling the 
-year cumulative incidence of risk 2 under a pre-determined level 
, 
. For simplicity, we only consider linear decision rules in this work, i.e., 
. Then the restricted optimal linear decision rule is equivalent to finding 
, such that
 |
(2.2) |
with the constraint 
, where 
.
Let 
be the unrestricted estimators that minimize the 
-year cumulative incidence of risk 
, 
. We have the following relation:
 |
If 
, 
is a null set and there is no solution for 
; if 
, we have 
because 
. Otherwise, the restricted optimal treatment regime needs to be searched in the set 
.
3. Estimation procedure
To estimate the restricted optimal treatment regime, we make the following three assumptions, as commonly used in the causal inference literature (Rubin, 1974). (i) The stable unit treatment value assumption: 
, 
and 
. (ii) The no unmeasured confounder assumption: given covariates 
, treatment assignment 
is independent of potential event times 
, 
, and potential censoring times 
, for 
. (iii) The assumptions for event and censoring times: given covariates 
and treatment assignment 
, the event times 
, 
are independent of 
, and the censoring time 
is independent of 
, 
, 
and 
.
Based on the above assumptions, we have
 |
where 
is the conditional CIF of risk 
defined based on the observed data. The relatively restrictive censoring assumption that 
is independent of all other variables is needed to derive a simple, model-free estimator for the regime-specific CIFs, which will be introduced shortly. A similar assumption was adopted in Jiang and others (2017a) for deriving estimators of the regime-specific survival function. Such an assumption can be relaxed; however, a model needs to be assumed for censoring times, and the inverse probability of the censoring-weighted technique needs to be used for constructing the estimator for the regime-specific CIFs.
Next, we propose consistent estimators for the CIFs based on the definition in (2.1). Specifically, a consistent estimator for 
is given by
 |
(3.3) |
where 
and 
are consistent estimators for 
and 
. Here 
and 
.
Define the counting process of risk 
as 
for 
and let 
. Moreover, let 
denote the at-risk process for subject 
. Under the assumed independent censoring assumption, Jiang and others (2017a) proposed an inverse propensity score-weighted Kaplan–Meier estimator for the survival function under regime 
, i.e.,
 |
where
 |
and 
is an estimator of the propensity score 
. It can be shown that as long as 
is a consistent estimator of 
, 
is a consistent estimator of the overall survival function 
. In practice, the propensity score is either known by design, as in randomized clinical trials, or estimated from data based on a posited parametric model, such as a logistic regression model, or estimated nonparametrically using a kernel or tree regression.
Similarly, we propose a consistent estimator of 
as
 |
Then, a consistent estimator 
of 
can be obtained based on (3.3).
Given the consistent estimator 
, 
, a natural estimator of the proposed restricted optimal treatment regime can be obtained by minimizing 
subject to the constraint 
. However, such a restricted optimization problem may not be easy to solve. Here, we propose an approximated solution by penalization. Specifically, we define the approximated solution for 
as
 |
(3.4) |
where 
is a large number, e.g., 
, and 
. Note that when 
, no penalization is added; however, when 
, a large penalty is added to the target function 
to encourage 
to satisfy the constraint. It can be seen that as 
, the penalized optimization problem will become the original restricted optimization problem.
The penalized optimization problem defined in (3.4) may still be challenging because the estimators 
are not smooth functions of 
, and the resulted solution may be trapped in local minima. Following Jiang and others (2017a), to reduce the bias due to discreteness, we apply kernel smoothing by replacing 
with 
in the computation of 
to obtain the smoothed estimators 
, 
, where 
is the cumulative distribution function of the standard normal distribution, and 
is a bandwidth parameter that goes to 0 as 
. As suggested in Jiang and others (2017a), we choose 
, where 
. Under such a rate, the estimated value functions of the original indicator regime and its smoothed approximation are asymptotically equivalent.
After smoothing, the objective function can be optimized directly using standard software, such as the “optim” function in R. Because the objective function is not convex, the solution may still be sensitive to the initial values. In practice, we suggest to trying a sequence of different initial values with 
, such as the unit vectors with one of the elements as 1 and the others as 0.
4. Simulation study
In our simulations, we consider two covariates, 
and 
, which are generated independently from the uniform distribution 
. The treatment indicator 
is generated from a logistic regression model with covariates 
and 
. We consider two cases for the propensity score: 
(case 1) or 
(case 2). In addition, we consider a proportional hazards model for the CIF of risk 1 (Fine and Gray, 1999):
 |
where 
is a predetermined constant that controls the proportion of risk 1, i.e., 
. Given 
, the conditional cumulative distribution function for the survival time of risk 1 can be derived as
 |
Given 
, the conditional cumulative distribution function for the survival time of risk 2 is chosen as the exponential with rate 
, i.e.,
 |
Then the CIF for risk 2 is given by
 |
To generate competing risk event times, we first generate 
(
or 2) from a Bernoulli distribution with the success probability 
. Then, we generate time 
by
 |
where 
is generated from the uniform distribution 
. The censoring time 
is generated from the uniform distribution 
, where 
is chosen to yield 15% and 40% censoring proportions.
We consider three values for 
: 0.2, 0.5, and 0.8, which give respectively about 35%, 62%, and 76% risk 1 rates (i.e. the percentage of subjects for whom 
and 
) under the 15% censoring rate and about 28%, 47%, and 55% risk 1 rates under the 40% censoring rate. Moreover, we set 
. Because of the proportional hazards model formulation for the CIF of risk 1, the unrestricted optimal treatment regime for risk 1, i.e., 
that minimizes 
, is independent of the value of 
and 
and is given by 
under the norm-1 constraint. However, the unrestricted optimal treatment regime for risk 2 is very complicated and not linear. Here, we use the grid search method to find the unrestricted optimal linear decision rule for risk 2, i.e., 
that minimizes 
, which is dependent on the value of 
. The resulting true parameter values of 
and 
, and their associated cumulative incidence values 
, 
, are reported (see Table S1 of supplementary material available at Biostatistics online). In addition, a set of 
values are selected based on the two cut-off points, 
and 
, that satisfy 
. The first two values are within the range, whereas the last value is greater than 
. The true parameter 
in the proposed restricted optimal linear decision rule for different 
and 
values are also obtained using the grid search method and reported in the truth table in the supplementary material available at Biostatistics online. It can be seen that when 
, which corresponds to the last column of the table, the restricted optimal linear decision rule becomes the unrestricted optimal linear decision rule, i.e., 
.
When estimating the restricted optimal treatment regime using the proposed method, we consider two propensity score fittings: the true propensity score (denoted by “true score”) and a standard logistic regression fit with 
and 
included as predictors (denoted by “logistic”). Therefore, for case 1, the fitted logistic regression model for the propensity score is correctly specified, whereas for case 2, it is misspecified.
For each setting, we consider sample size 
and run 
replications. The simulation results for 
and 0.5 are presented in Tables 1 and 2, and the results for the setting with 
are similar to Tables 1 and 2 (see Table S2 of supplementary material available at Biostatistics online). The following values are reported: (i) the mean and empirical standard deviation (SD) of the estimated 
coefficients; (ii) the mean and SD of the estimated 
-year cumulative incidence of risk 1, 
, under the estimated restricted optimal treatment regime; (iii) the mean and SD of the true 
-year cumulative incidence of risk 1, 
, under the estimated restricted optimal treatment regime computed using the Monte Carlo method based on a large simulated dataset; (iv) the mean and SD of the proportion of making the correct decision (PCD) when comparing the estimated restricted optimal treatment regime with the true restricted optimal treatment regime; and (v) the proportion of the estimated 
-year cumulative incidence of risk 2, 
, under the estimated restricted optimal treatment regime being controlled under the pre-determined level 
.
Table 1.
Simulation results (
, 
)
| Cens |
|
Parameter | Truth | True score | Logistic | True score | Logistic |
|---|---|---|---|---|---|---|---|
| Case 1 | Case 2 | ||||||
| 15% | 0.3 |
|
0.66 | 0.58(0.165) | 0.58(0.160) | 0.52(0.195) | 0.55(0.163) |
|
 0.05 |
 0.08(0.250) |
 0.05(0.234) |
 0.20(0.273) |
 0.03(0.250) |
||
|
0.75 | 0.74(0.129) | 0.75(0.122) | 0.74(0.163) | 0.76(0.133) | ||
|
0.16 | 0.16(0.030) | 0.16(0.029) | 0.15(0.030) | 0.16(0.024) | ||
|
0.16 | 0.19(0.017) | 0.18(0.017) | 0.19(0.021) | 0.19(0.020) | ||
| PCD | — | 0.86(0.108) | 0.85(0.113) | 0.87(0.099) | 0.87(0.097) | ||
|
— | 1 | 0.996 | 1 | 1 | ||
| 0.4 |
|
0.3 | 0.31(0.231) | 0.31(0.216) | 0.25(0.249) | 0.25(0.248) | |
|
 0.53 |
 0.45(0.192) |
 0.45(0.187) |
 0.46(0.232) |
 0.31(0.198) |
||
|
0.79 | 0.77(0.119) | 0.78(0.114) | 0.77(0.138) | 0.85(0.121) | ||
|
0.14 | 0.13(0.026) | 0.13(0.026) | 0.13(0.029) | 0.14(0.025) | ||
|
0.14 | 0.15(0.009) | 0.15(0.008) | 0.15(0.013) | 0.15(0.009) | ||
| PCD | — | 0.92(0.073) | 0.93(0.074) | 0.91(0.088) | 0.90(0.084) | ||
|
— | 1 | 1 | 1 | 1 | ||
| 0.5 |
|
0 | 0.08(0.261) | 0.08(0.255) | 0.08(0.295) | 0.17(0.294) | |
|
 0.71 |
 0.62(0.181) |
 0.62(0.178) |
 0.57(0.230) |
 0.37(0.220) |
||
|
0.71 | 0.70(0.142) | 0.70(0.134) | 0.71(0.150) | 0.82(0.154) | ||
|
0.13 | 0.13(0.027) | 0.13(0.027) | 0.12(0.031) | 0.14(0.026) | ||
|
0.13 | 0.14(0.009) | 0.14(0.007) | 0.14(0.010) | 0.15(0.010) | ||
| PCD | — | 0.86(0.114) | 0.86(0.112) | 0.85(0.118) | 0.81(0.099) | ||
|
— | 1 | 1 | 1 | 1 | ||
| 40% | 0.3 |
|
0.66 | 0.57(0.191) | 0.57(0.181) | 0.49(0.218) | 0.54(0.183) |
|
 0.05 |
 0.08(0.279) |
 0.06(0.264) |
 0.19(0.290) |
 0.04(0.273) |
||
|
0.75 | 0.73(0.144) | 0.74(0.128) | 0.76(0.151) | 0.76(0.159) | ||
|
0.16 | 0.16(0.034) | 0.16(0.033) | 0.15(0.034) | 0.16(0.028) | ||
|
0.16 | 0.19(0.019) | 0.19(0.019) | 0.19(0.019) | 0.19(0.021) | ||
| PCD | — | 0.85(0.111) | 0.85(0.115) | 0.85(0.101) | 0.85(0.100) | ||
|
— | 1 | 1 | 0.998 | 1 | ||
| 0.4 |
|
0.3 | 0.31(0.257) | 0.32(0.236) | 0.25(0.270) | 0.25(0.278) | |
|
 0.53 |
 0.43(0.204) |
 0.44(0.202) |
 0.46(0.236) |
 0.30(0.223) |
||
|
0.79 | 0.77(0.132) | 0.77(0.121) | 0.76(0.155) | 0.84(0.147) | ||
|
0.14 | 0.13(0.028) | 0.13(0.028) | 0.13(0.032) | 0.14(0.028) | ||
|
0.14 | 0.15(0.010) | 0.15(0.010) | 0.15(0.014) | 0.15(0.011) | ||
| PCD | — | 0.92(0.075) | 0.92(0.074) | 0.90(0.083) | 0.88(0.094) | ||
|
— | 1 | 1 | 0.998 | 1 | ||
| 0.5 |
|
0 | 0.09(0.285) | 0.09(0.278) | 0.08(0.314) | 0.16(0.314) | |
|
 0.71 |
 0.60(0.195) |
 0.60(0.200) |
 0.55(0.238) |
 0.37(0.240) |
||
|
0.71 | 0.70(0.154) | 0.70(0.146) | 0.71(0.164) | 0.80(0.189) | ||
|
0.13 | 0.12(0.029) | 0.12(0.028) | 0.12(0.033) | 0.14(0.029) | ||
|
0.13 | 0.14(0.008) | 0.14(0.009) | 0.14(0.011) | 0.15(0.011) | ||
| PCD | — | 0.85(0.114) | 0.85(0.116) | 0.84(0.115) | 0.81(0.109) | ||
|
— | 1 | 1 | 1 | 1 | ||
Table 2.
Simulation results 
| Cens |
|
Parameter | Truth | True score | Logistic | True score | Logistic |
|---|---|---|---|---|---|---|---|
| Case 1 | Case 2 | ||||||
| 15% | 0.2 |
|
0.61 | 0.50(0.201) | 0.51(0.189) | 0.42(0.247) | 0.40(0.225) |
|
 0.1 |
 0.14(0.271) |
 0.12(0.254) |
 0.26(0.295) |
 0.17(0.238) |
||
|
0.79 | 0.77(0.144) | 0.78(0.137) | 0.76(0.180) | 0.82(0.138) | ||
|
0.35 | 0.35(0.037) | 0.35(0.036) | 0.34(0.041) | 0.36(0.031) | ||
|
0.35 | 0.38(0.018) | 0.38(0.017) | 0.39(0.032) | 0.38(0.021) | ||
| PCD | — | 0.89(0.055) | 0.90(0.051) | 0.87(0.075) | 0.90(0.046) | ||
|
— | 0.998 | 1 | 0.996 | 1 | ||
| 0.3 |
|
0.16 | 0.21(0.228) | 0.20(0.220) | 0.17(0.279) | 0.16(0.259) | |
|
 0.6 |
 0.52(0.190) |
 0.54(0.175) |
 0.50(0.233) |
 0.35(0.182) |
||
|
0.78 | 0.76(0.135) | 0.76(0.122) | 0.75(0.159) | 0.86(0.135) | ||
|
0.31 | 0.31(0.034) | 0.31(0.034) | 0.31(0.049) | 0.34(0.036) | ||
|
0.31 | 0.32(0.011) | 0.32(0.009) | 0.33(0.023) | 0.33(0.011) | ||
| PCD | — | 0.90(0.049) | 0.90(0.048) | 0.88(0.071) | 0.88(0.045) | ||
|
— | 1 | 1 | 1 | 1 | ||
| 0.4 |
|
0 | 0.02(0.232) | 0.02(0.223) | 0.06(0.275) | 0.15(0.266) | |
|
 0.71 |
 0.66(0.173) |
 0.66(0.158) |
 0.58(0.224) |
 0.36(0.192) |
||
|
0.71 | 0.68(0.160) | 0.68(0.149) | 0.71(0.166) | 0.85(0.155) | ||
|
0.31 | 0.30(0.037) | 0.30(0.036) | 0.30(0.056) | 0.34(0.036) | ||
|
0.31 | 0.31(0.008) | 0.31(0.007) | 0.32(0.013) | 0.32(0.011) | ||
| PCD | — | 0.90(0.049) | 0.91(0.047) | 0.88(0.061) | 0.83(0.051) | ||
|
— | 1 | 1 | 1 | 1 | ||
| 40% | 0.2 |
|
0.61 | 0.46(0.250) | 0.46(0.241) | 0.38(0.284) | 0.38(0.262) |
|
 0.1 |
 0.15(0.299) |
 0.15(0.297) |
 0.28(0.315) |
 0.18(0.280) |
||
|
0.79 | 0.76(0.176) | 0.77(0.163) | 0.74(0.213) | 0.80(0.183) | ||
|
0.35 | 0.34(0.046) | 0.34(0.045) | 0.34(0.046) | 0.35(0.040) | ||
|
0.35 | 0.39(0.024) | 0.38(0.023) | 0.39(0.032) | 0.38(0.027) | ||
| PCD | — | 0.88(0.067) | 0.88(0.061) | 0.86(0.077) | 0.89(0.057) | ||
|
— | 1 | 0.998 | 0.998 | 0.998 | ||
| 0.3 |
|
0.16 | 0.21(0.274) | 0.19(0.269) | 0.15(0.311) | 0.16(0.285) | |
|
 0.6 |
 0.49(0.227) |
 0.49(0.228) |
 0.51(0.253) |
 0.36(0.242) |
||
|
0.78 | 0.75(0.168) | 0.76(0.156) | 0.72(0.196) | 0.81(0.212) | ||
|
0.31 | 0.30(0.043) | 0.30(0.043) | 0.30(0.054) | 0.34(0.044) | ||
|
0.31 | 0.33(0.016) | 0.32(0.015) | 0.33(0.023) | 0.33(0.015) | ||
| PCD | — | 0.88(0.062) | 0.88(0.059) | 0.87(0.072) | 0.87(0.057) | ||
|
— | 1 | 1 | 1 | 1 | ||
| 0.4 |
|
0 | 0.04(0.274) | 0.03(0.274) | 0.03(0.306) | 0.14(0.293) | |
|
 0.71 |
 0.63(0.206) |
 0.64(0.197) |
 0.59(0.233) |
 0.39(0.251) |
||
|
0.71 | 0.67(0.189) | 0.67(0.188) | 0.68(0.197) | 0.79(0.238) | ||
|
0.31 | 0.29(0.045) | 0.29(0.045) | 0.29(0.059) | 0.33(0.044) | ||
|
0.31 | 0.32(0.011) | 0.32(0.011) | 0.32(0.014) | 0.33(0.015) | ||
| PCD | — | 0.88(0.059) | 0.88(0.060) | 0.87(0.063) | 0.82(0.061) | ||
|
— | 1 | 1 | 1 | 1 | ||
From the results, we can see that in most scenarios, the estimated 
coefficients and the estimated and true 
-year cumulative incidence of risk 1 under the estimated optimal treatment regime are close to their true values. The PCD is relatively high, ranging from 80% to 90%, and it also increases as the sample size increases and the censoring proportion decreases. In addition, the proportion of the estimated 
-year cumulative incidence of risk 2 under the estimated restricted optimal treatment regime being controlled under the pre-determined level 
is close to 1 in all of the settings. These results show that the restricted optimal treatment regime obtained by the proposed method can minimize the 
-year cumulative incidence of risk 1 while controlling the 
-year cumulative incidence of risk 2 under a predetermined level, as designed.
Comparing the two estimated restricted optimal treatment regimes obtained based on true propensity scores and the estimated propensity scores using logistic regression, we find that the results are generally comparable for both cases 1 and 2, especially for small 
values. When 
is large, the estimated 
coefficients based on the misspecified logistic regression of the propensity score under case 2 have relatively larger biases compared with the estimators based on true scores, and the corresponding estimated restricted optimal treatment regimes have slightly smaller PCDs. This suggests that the proposed estimation method has reasonable performance to the misspecification of the propensity score model, especially in terms of treatment decision.
It is noticed that there exists the bias in the estimated 
coefficients although the bias in the estimated 
coefficients gets smaller as the sample size increases and the censoring proportion decreases. The bias in the estimated 
coefficients may come from two aspects. First, it may be due to the smooth approximation of the indicator function by the Gaussian CDF in the treatment rule. Second, we consider a penalized estimation in our implementation, which is an approximation to the original constrained optimization problem. Theoretically, the penalty 
should go to infinity. However, in practice, we choose a large value for 
.
5. Application to HIV data
We apply the proposed method to an HIV dataset obtained from an observational study collected by HSSC. Two combinations of HIV treatment are considered: “PIs+NRTIs” (regime 1) and “NNRTIs+NRTIs” (regime 2). A total of 229 HIV patients are included in this study. Among them, 171 patients are assigned to regime 1 and 58 patients to regime 2. In addition, 77 (33.6%) patients have either CD4 counts drop below 500 cells/
or HIV viral loads greater than 200 copies/mL (referred to as “risk 1”), and 105 (45.9%) patients have serious drug-induced side effects such as liver damage, kidney problems, or depression (referred to as “risk 2”).
The survival time of interest is defined as days after the first drug administration on record to the first onset of either risk, whichever comes first. The risk-type indicator is recorded as 1 for “risk 1” and 2 for “risk 2”. If neither risk occurred during the study period, the survival time is censored and indicated by 0 for a risk-type indicator. The baseline characteristic for an HIV patient includes standardized age, gender (male 
1 or female 
0), race (black 
1 or other 
0), insurance type (government/commercial (GC) 
1 or other 
0), region (urban 
1 or rural 
0), and marital status (single, married, or other). Among them, 56.8% of the patients are males, 76.4% are Black, and most of them have GC insurance (96.1%) and live in an urban area (85.2%). More than half (68.6%) are single, and only 14.8% are married.
First, we fit propensity scores through the logistic regression with all the demographic variables included as linear form and use the LASSO penalty to select the important covariates. This can be done using the“cv.glmnet” function in the “glmnet” package (Friedman and others, 2010) in R. As a result, all covariates, except gender, are kept in the model for propensity scores. We kept all covariates in the survival model for estimating the CIFs of the two risks.
We consider estimation of the restricted optimal treatment regime at 1, 2, 3, and 4 years. That is, time 
, 730, 1095, and 1460 days after drug administration. In Table 3, we report the estimated coefficients 
and 
in the unrestricted optimal treatment regime, which minimizes the 
-year cumulative incidence of risk 1 and risk 2, respectively. Then, based on the range from 
to 
, we select 
as a common value that is included in this range and report the corresponding estimated coefficients 
in the restricted optimal treatment regime. It can be seen that the 
-year cumulative incidences of risk 2 under the estimated restricted optimal treatment regime are all controlled at level 
as desired; however, the 
-year cumulative incidence of risk 2 ranges between 0.379 and 0.629 under the estimated unrestricted optimal treatment regime that minimizes the 
-year cumulative incidence of risk 1. In addition, with the constraint on the cumulative incidence of risk 2, the estimated restricted optimal treatment regime increases the cumulative incidence of risk 1 compared with the estimated unrestricted optimal treatment regime. The magnitude of inflation depends on how stringent the posited constraint is on risk 2. For example, when 
, 
and the cumulative incidence of risk 1 stays the same at 0.203, whereas when 
, 
and the cumulative incidence of risk 1 increases from 0.273 to 0.428.
Table 3.
Estimated regimes for HIV study
One year ( ) |
Two years ( ) |
|||||
|---|---|---|---|---|---|---|
|
|
|
|
|
|
|
| Intercept | 0.194 | -0.324 | 0.108 | 0.762 | -0.645 | 0.549 |
| Age | 0.252 | 0.130 | 0.256 | -0.032 | 0.110 | 0.176 |
| Race | -0.329 | 0.427 | -0.342 | -0.480 | 0.377 | -0.457 |
| Insurance | 0.039 | 0.694 | 0.066 | -0.342 | 0.584 | 0.169 |
| Single vs other | 0.856 | -0.043 | 0.859 | 0.182 | 0.212 | 0.560 |
| Married vs other | -0.031 | -0.306 | -0.027 | 0.026 | -0.040 | -0.199 |
| Region | -0.023 | -0.017 | 0.050 | -0.100 | 0.027 | -0.275 |
| Gender | -0.236 | -0.344 | -0.245 | -0.164 | -0.204 | 0.021 |
|
0.203 | 0.375 | 0.203 | 0.237 | 0.441 | 0.289 |
|
0.379 | 0.243 | 0.379 | 0.531 | 0.297 | 0.400 |
Three years ( ) |
Four years ( ) |
|||||
|---|---|---|---|---|---|---|
|
|
|
|
|
|
|
| Intercept | 0.799 | -0.518 | 0.548 | 0.771 | -0.203 | -0.143 |
| Age | 0.038 | -0.177 | 0.365 | 0.062 | -0.170 | 0.284 |
| Race | -0.382 | 0.237 | -0.210 | -0.429 | 0.191 | 0.120 |
| Insurance | -0.353 | 0.717 | -0.077 | -0.362 | 0.676 | 0.398 |
| Single vs other | 0.057 | 0.032 | 0.647 | 0.108 | -0.300 | 0.478 |
| Married vs other | 0.145 | -0.214 | -0.138 | 0.130 | -0.520 | -0.420 |
| Region | -0.224 | -0.128 | -0.170 | -0.159 | -0.065 | 0.397 |
| Gender | -0.128 | 0.256 | 0.224 | -0.179 | 0.268 | -0.405 |
|
0.248 | 0.512 | 0.404 | 0.273 | 0.534 | 0.428 |
|
0.600 | 0.309 | 0.400 | 0.629 | 0.323 | 0.400 |
We also compare the number of patients assigned to the two treatment regimes with the actual treatment received (see Table S3 of supplementary material available at Biostatistics online). The proportions where the treatment dictated by the estimated optimal treatment regime is consistent with the received treatment (cons%) are also reported. Patients received regime 1 more frequently than regime 2 in our dataset (171 vs 58) because PIs are considered to have slightly greater CD4 cell count recovery and lower antiretroviral drug resistance evolution with virologic failure in practice (World Health Organization, 2016). Similar arguments about allocations of these two drugs are discussed in the literature (Jiang and others, 2017b). However, the picture can be quite different when we consider side effects and apply restricted optimal treatment regimes. The estimated unrestricted optimal treatment regime that minimizes the 
-year cumulative incidence of risk 1 is found to be closer to the actual treatment assignment compared with the estimated unrestricted optimal treatment regime that minimizes the 
-year cumulative incidence of risk 2 and the estimated restricted optimal treatment regimes. This also suggests that doctors are likely to assign drugs with direct treatment effect.
In practice, the optimal decision should be provided through combining the experts’ experience and statistical methods. Therefore, we provide the estimation of the restricted optimal treatment regime with a sequence of 
values in the range from 
to 
and plot the number of patients assigned to each treatment by the estimated restricted optimal treatment regime in Figure 2. In the plot, the dashed vertical line corresponds to 
, beyond which the restricted optimal treatment regime is equivalent to the unrestricted optimal treatment regime. By comparing the two curves, we can see how treatments are distributed among patients under the restricted optimal treatment regime for different 
values. Based on the profile of treatment decision rule (Figure 2) with respect to the different level of risk at different 
year of interest, the practitioner can choose which strategy should be used in practice that will be a balance between the clinical benefit and risk control. It is worthwhile pointing out that, due to the limitations of the electronic dataset from HSSC, we do not have complete information about the laboratory measure of CD4 or viral load before the first medication use, which can be used as a measure of disease severity in HIV patients and should be considered in assigning the treatment. For illustration purposes, we only use all demographic variables without adjusting disease severity. If more information about disease severity is available, the proposed method will provide more accurate assignment of treatment.
Fig. 2.
Treatment distribution for HIV data
6. Discussion
In this work, we propose a new method for estimating the restricted optimal treatment regime for competing risks data, which minimizes the cumulative incidence of the primary risk at a fixed time point while controlling the corresponding cumulative incidence of the second risk under a pre-specified level. A penalization method is developed for obtaining an approximate solution for the challenging restricted optimization problem. In the proposed method, we only consider the inverse propensity score-weighted estimators for the regime-specific overall survival function and CIFs. However, it can be extended to accommodate the augmented inverse propensity score-weighted estimators, similar to what is described in Jiang and others (2017a). Such an extension requires modeling and estimation of the CIFs of both risks simultaneously, for example, as in Lu and Peng (2008) and Mao and Lin (2017), which may be difficult to implement in practice. In addition, the main interest of our current project is to find the optimal treatment regime based on patients’ first occurrence of treatment failure or serious side effects. The proposed method can be extended to derive the restricted optimal dynamic treatment regime for multiple treatment decision time points such as finding the dynamic optimal regimes based on the repeated measures of viral load, CD4, and side effects, which can be studied in the future.
It is “ideal” to maximize the value function of the primary clinical outcome while controlling the individual risk at a certain level for the risk-associated outcome. For example, we may consider the following restricted optimization problem:
 |
where 
is an estimator of the CIF of the risk-associated outcome for subject 
given the treatment regime 
and can be obtained based on the proportional hazards regression as studied in Fine and Gray (1999). However, such a restricted optimization problem may not be able to be implemented reliably in practice. We have done simulations under the same settings as Case 1 in the article. We found that when 
is relatively small, the proportion by which the above-restricted optimization problem can be solved may be low. In addition, even when there is a solution, the estimated 
cannot always be controlled at the desired risk level.
The inference of 
from the value search is a challenge because the inference of the value search estimator, even without the constraint on average risk, is a challenging problem. As studied in Shi and others (2018), the convergence rate of the value search estimator is 
, and its asymptotic distribution is not normal. For the considered constrained value search estimator, its asymptotic properties are even harder to study. This is an interesting problem and will be studied in the future.
Supplementary Material
Acknowledgments
The project is supported by an award to Drs. Xiaoming Li and Jiajia Zhang from the National Institute of Allergy and Infectious Diseases (NIAID)(1R01AI127203-01A1). The content is solely the responsibility of the authors and does not necessarily represent the official views of NIAID.
Conflict of Interest: None declared.
References
- Bai, X., Tsiatis, A. A., Lu, W. and Song, R. (2017). Optimal treatment regimes for survival endpoints using a locally-efficient doubly-robust estimator from a classification perspective. Lifetime Data Analysis 23, 585–604. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Borges, Á. H., Lundh, A., Tendal, B., Bartlett, J. A., Clumeck, N., Costagliola, D., Daar, E. S., Echeverría, P., Gisslén, M., Huedo-Medina, T. B.. and others. (2016). Nonnucleoside reverse-transcriptase inhibitor- vs ritonavir-boosted protease inhibitor–based regimens for initial treatment of HIV infection: a systematic review and metaanalysis of randomized trials. Clinical Infectious Diseases 63, 268–280. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Dybul, M., Fauci, A. S., Bartlett, J. G., Kaplan, J. E. and Pau, A. K. (2002). Guidelines for using antiretroviral agents among HIV-infected adults and adolescents. recommendations of the panel on clinical practices for treatment of HIV. Annals of Internal Medicine 137, 381–433. [DOI] [PubMed] [Google Scholar]
- Fine, J. P. and Gray, R. J. (1999). A proportional hazards model for the subdistribution of a competing risk. Journal of the American Statistical Association 94, 496–509. [Google Scholar]
- Friedman, J., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33, 1. [PMC free article] [PubMed] [Google Scholar]
- Gray, R. J. (1988). A class of k-sample tests for comparing the cumulative incidence of a competing risk. The Annals of Statistics 16, 1141–1154. [Google Scholar]
- Günthard, H. F., Saag, M. S., Benson, C. A., Del Rio, C., Eron, J. J., Gallant, J. E., Hoy, J. F., Mugavero, M. J., Sax, P. E., Thompson, M. A.. and others. (2016). Antiretroviral drugs for treatment and prevention of HIV infection in adults: 2016 recommendations of the international antiviral society–usa panel. JAMA 316, 191–210. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Haubrich, R. H., Riddler, S. A., DiRienzo, A. G., Komarow, L., Powderly, W. G., Klingman, K., Garren, K. W., Butcher, D. L., Rooney, J. F., Haas, D. W.. and others. (2009). Metabolic outcomes in a randomized trial of nucleoside, nonnucleoside and protease inhibitor-sparing regimens for initial HIV treatment. AIDS (London, England) 23, 1109. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Health and Human Services (2017). Limitations to Treatment Safety and Efficacy (posted October17, 2017) https://aidsinfo.nih.gov/guidelines/html/1/adult-and-adolescent-arv/31/adverse-effects-of-arv.
- Jiang, R., Lu, W., Song, R. and Davidian, M. (2017a). On estimation of optimal treatment regimes for maximizing t-year survival probability. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 79, 1165–1185. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Jiang, R., Lu, W., Song, R., Hudgens, M. and Naprvavnik, S. (2017b). Doubly robust estimation of optimal treatment regimes for survival data—with application to an HIV/AIDS study. The Annals of Applied Statistics 11, 1763–1786. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Klein, J. P. and Andersen, P. K. (2005). Regression modeling of competing risks data based on pseudovalues of the cumulative incidence function. Biometrics 61, 223–229. [DOI] [PubMed] [Google Scholar]
- Lu, W. and Peng, L. (2008). Semiparametric analysis of mixture regression models with competing risks data. Lifetime Data Analysis 14, 231–252. [DOI] [PubMed] [Google Scholar]
- Mao, L. and Lin, D. Y. (2017). Efficient estimation of semiparametric transformation models for the cumulative incidence of competing risks. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 79, 573–587. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Murphy, S. A. (2003). Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 65, 331–355. [Google Scholar]
- Murphy, S. A. (2005). An experimental design for the development of adaptive treatment strategies. Statistics in Medicine 24, 1455–1481. [DOI] [PubMed] [Google Scholar]
- Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66, 688–701. [Google Scholar]
- Shi, C., Lu, W. and Song, R. (2018). A massive data framework for m-estimators with cubic-rate. Journal of the American Statistical Association 113, 1698–1709. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Simpson, K. N., Hanson, K. A., Harding, G., Haider, S., Tawadrous, M., Khachatryan, A., Pashos, C. L. and Wu, A. W. (2014). Review of the impact of NNRTI-based HIV treatment regimens on patient-reported disease burden. AIDS Care 26, 466–475. [DOI] [PubMed] [Google Scholar]
- Smith, K. Y., Patel, P., Fine, D., Bellos, N., Sloan, L., Lackey, P., Kumar, P. N., Sutherland-Phillips, D. H., Vavro, C., Yau, L.. and others. (2009). Randomized, double-blind, placebo-matched, multicenter trial of abacavir/lamivudine or tenofovir/emtricitabine with lopinavir/ritonavir for initial HIV treatment. AIDS 23, 1547–1556. [DOI] [PubMed] [Google Scholar]
- Staszewski, S., Morales-Ramirez, J., Tashima, K. T., Rachlis, A., Skiest, D., Stanford, J., Stryker, R., Johnson, P., Labriola, D. F., Farina, D.. and others. (1999). Efavirenz plus zidovudine and lamivudine, efavirenz plus indinavir, and indinavir plus zidovudine and lamivudine in the treatment of HIV-1 infection in adults. New England Journal of Medicine 341, 1865–1873. [DOI] [PubMed] [Google Scholar]
- Sun, L., Liu, J., Sun, J. and Zhang, M.-J. (2006). Modeling the subdistribution of a competing risk. Statistica Sinica 16, 1367–1385. [Google Scholar]
- Wang, Y., Fu, H. and Zeng, D. (2018). Learning optimal personalized treatment rules in consideration of benefit and risk: with an application to treating type 2 diabetes patients with insulin therapies. Journal of the American Statistical Association 113, 1–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Watkins, C. J. C. H. and Dayan, P. (1992). Q-learning. Machine Learning 8, 279–292. [Google Scholar]
- World Health Organization (2016). Consolidated Guidelines on the Use of Antiretroviral Drugs for Treating and Preventing HIV Infection: Recommendations for a Public Health Approach. Switzerland: World Health Organization. [PubMed] [Google Scholar]
- Worm, S. W., Sabin, C., Weber, R., Reiss, P., El-Sadr, W., Dabis, F., De Wit, S., Law, M., Monforte, A. D., Friis-Møller, N.. and others. (2010). Risk of myocardial infarction in patients with HIV infection exposed to specific individual antiretroviral drugs from the 3 major drug classes: the data collection on adverse events of anti-HIV drugs (D:A:D) study. The Journal of Infectious Diseases 201, 318–330. [DOI] [PubMed] [Google Scholar]
- Yavuz, I., Chng, Y. and Wahed, A. S. (2018). Estimating the cumulative incidence function of dynamic treatment regimes. Journal of the Royal Statistical Society: Series A (Statistics in Society) 181, 85–106. [Google Scholar]
- Zhang, B., Tsiatis, A. A., Laber, E. B. and Davidian, M. (2012). A robust method for estimating optimal treatment regimes. Biometrics 68, 1010–1018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zhang, B., Tsiatis, A. A., Laber, E. B. and Davidian, M. (2013). Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions. Biometrika 100, 681–694. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zhao, Y., Zeng, D., Laber, E. B. and Kosorok, M. R. (2015a). New statistical learning methods for estimating optimal dynamic treatment regimes. Journal of the American Statistical Association 110, 583–598. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zhao, Y., Zeng, D., Laber, E. B., Song, R. and Kosorok, M. R. (2015b). Doubly robust learning for estimating individualized treatment with censored data. Biometrika 102, 151–168. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zhao, Y., Zeng, D., Rush, A. J. and Kosorok, M. R. (2012). Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association 107, 1106–1118. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.