Inference about the expected performance of a data-driven dynamic treatment regime

Abstract

Background

A dynamic treatment regime (DTR) comprises a sequence of decision rules, one per stage of intervention, that recommends how to individualize treatment to patients based on evolving treatment and covariate history. These regimes are useful for managing chronic disorders, and fit into the larger paradigm of personalized medicine. The Value of a DTR is the expected outcome when the DTR is used to assign treatments to a population of interest.

Purpose

The Value of a data-driven DTR, estimated using data from a sequential multiple assignment randomized trial, is both a data-dependent parameter and a non-smooth function of the underlying generative distribution. These features introduce additional variability that is not accounted for by standard methods for conducting statistical inference, e.g., the bootstrap or normal approximations, if applied without adjustment. Our purpose is to provide a feasible method for constructing valid confidence intervals for this quantity of practical interest.

Methods

We propose a conceptually simple and computationally feasible method for constructing valid confidence intervals for the Value of an estimated DTR based on subsampling. The method is self-tuning by virtue of an approach called the double bootstrap. We demonstrate the proposed method using a series of simulated experiments.

Results

The proposed method offers considerable improvement in terms of coverage rates of the confidence intervals over the standard bootstrap approach.

Limitations

In this paper, we have restricted our attention to Q-learning for estimating the optimal DTR. However, other methods can be employed for this purpose; to keep the discussion focused, we have not explored these alternatives.

Conclusions

Subsampling-based confidence intervals provide much better performance compared to standard bootstrap for the Value of an estimated DTR.

1 Introduction

Dynamic treatment regimes (DTRs) reflect the increasingly popular theme of personalized medicine in biostatistical research. DTRs facilitate personalized medicine in time-varying settings by allowing treatment selection to depend on time-varying – or dynamic – information. Operationally, a DTR consists of a sequence of decision rules, one per stage of clinical intervention, that recommends a treatment based on a patient's available treatment and covariate history. These decision rules offer a vehicle for the personalized management of many chronic conditions, e.g., alcohol and drug abuse [1], tobacco addiction [2], cancer [3, 4, 5, 6], HIV infection [7, 8], and mental illnesses [9, 10, 11], where a patient typically has to be treated at multiple stages. In essence, DTRs constitute operationalized clinical decision support systems, a key element of the chronic care model [12]; see Chakraborty and Moodie [13] for a book-length treatment of the topic of DTRs. A simple example of a two-stage behavioral DTR for smoking cessation is: “Initially provide a behavioral message with high degree of tailoring (message individually tailored according to the smoker's baseline variables), and provide a booster prevention message after six months as a follow-on intervention (while providing nicotine patch all along, to actively address the pharmacological aspect of smoking cessation).”

Expert opinion is one approach to constructing DTRs [e.g., 14, 15]; however, there has been a recent surge of interest in making DTRs evidence-based, i.e., data-driven. The quality of a DTR is usually assessed in terms of its Value. The Value of a DTR is the average primary outcome obtained when the DTR is applied to the entire population of interest (see below). A DTR is said to be optimal if it yields the highest Value. Most statistical research in the area of DTRs concerns: (a) the comparison of two or more preconceived DTRs in terms of their Value; or (b) the estimation of the optimal DTR, i.e., estimation of the sequence of decision rules that would result in the highest Value, within a certain class.

High-quality data (i.e., data free from causal confounding) for comparing or constructing DTRs can be obtained from Sequential Multiple Assignment Randomized Trials (SMARTs) [9, 16, 17, 18]. Methodological research on SMARTs is experiencing a steady growth to accommodate the increasing prevalence of such designs in practice, e.g., in cancer [6, 19, 20, 21, 22, 23], smoking [2], childhood autism [1, 24], childhood attention deficit hyperactivity disorder [25, 26], drug abuse during pregnancy [1, 27], and alcoholism [1]; see also [28] and [29] for comprehensive lists of SMART studies. The increasing popularity of SMARTs is further reflected in the recent NIH program announcements specifically asking for such designs [30]. For a discussion of SMART designs including power, efficiency, and sample sizes, see [17, 18, 25, 31, 32, 33, 34] and references therein. DTRs can also be constructed from longitudinal, observational studies; but the analysis becomes more complex in case of observational studies because the analytic techniques for such studies must proactively address selection bias and time-varying confounding [8, 35, 36]. In this article, for simplicity, we will restrict our attention to data from SMARTs only.

There exist a variety of methods for estimating DTRs from either SMART or observational data, including: G-estimation [37]; Q-learning [2, 38, 39]; marginal structural models [8, 10, 35]; outcome weighted learning [40]; and augmented inverse probability weighting [41, 42]. Regardless of the estimation method, accurate measures of uncertainty are needed if the estimated DTR will be used to inform clinical practice or guide subsequent research. We consider the problem of constructing a confidence interval for the Value of an estimated optimal DTR. This problem is made complex by the fact that the Value is a data-dependent and non-smooth parameter [37, 43]. Note that the Value of a fixed DTR (i.e., one that is not data-driven) does not suffer from these issues and has been addressed by numerous authors [e.g., 44, 45, 46, 47, 48]. We propose a conceptually simple and computationally feasible method for constructing a confidence interval for the Value of an estimated optimal DTR based on subsampling [e.g., 11, 49].

In Section 2 we review SMART designs. In Section 3 we define the Value of a DTR, and also describe Q-learning for estimating an optimal DTR. We introduce a novel subsampling based confidence interval in Section 4. The finite sample performance of the proposed confidence interval is evaluated using a suite of simulated experiments in Section 5. Section 6 provides a concluding discussion.

2 Setup and Notation

2.1 Sequential Multiple Assignment Randomized Trials

Initially, sequential randomization was only used as a conceptual tool to describe conditions needed to identify an optimal DTR from observational data [50, 51, 52, 53]. SMART trials, informed by this work, satisfy the sequential randomization condition by design [16, 17, 18]. SMART designs involve an initial randomization of patients to available treatment options, followed by re-randomizations at each subsequent stage of some or all of the patients to treatment options available at that stage. The re-randomizations and the set of treatment options at each stage may depend on information collected in prior stages such as how well the patient responded to the previous treatment.

Figure 1 shows a hypothetical SMART that may be used to estimate the behavioral DTR for smoking cessation discussed in the introduction. In this hypothetical trial, each participant is randomly assigned to one of two possible initial behavioral treatments: message with high degree of tailoring or message with low degree of tailoring (using baseline information). After six months, participants' intermediate outcomes (quit status, number of non-smoking months, number of cigarettes smoked while in the study, etc.) are collected, and participants are re-randomized to one of the two subsequent behavioral treatment options at stage 2: a booster prevention message, or a control message. All participants are provided with nicotine patches to address the pharmacological aspect of smoking cessation. One goal of the study is to construct a DTR that would maximize the mean number of non-smoking months over the 12-month study period. This hypothetical study is a simplified version of a real SMART as described in [2]; see also Strecher et al. [54] for detailed rationale behind considering such behavioral intervention strategies.

Figure 1.

Hypothetical SMART design schematic for the smoking cessation example. An “R” within a circle denotes randomization.

2.2 Data Structure

For clarity, we consider SMARTs with two stages of intervention only, as in Figure 1; generalization to more than two stages is relatively straightforward [55]. The longitudinal data trajectory observed on each patient has the form (O₁, A₁, O₂, A₂, O₃), where O_j, j = 1, 2 denotes the vector of covariates measured prior to treatment at the beginning of the j-th stage, O₃ denotes the outcomes measured at the end of stage 2 (end of the study), and A_j, j = 1, 2 denotes the treatment assigned at the j-th stage subsequent to observing O_j. The primary outcome is Y = r(O₁, A₁, O₂, A₂, O₃), for a known function r, which is observed at the end of the study. For example, in the smoking cessation study above, O₁ may include addiction severity and co-morbid conditions at baseline, O₂ may include the same variables observed at the end of stage 1 (six months post initial randomization) and adherence to initial treatment, and Y may be the number of non-smoking months over the 12-month study period. The methodology proposed in this article also applies to trial designs in which only a subset of subjects (e.g., nonresponders) are randomized at the second stage; this case is handled by a judicious choice of the outcome; see [55] for details and an example. We briefly illustrate this case via simulation in Section 5.

Define the history at each stage as H₁ = O₁ and $H_2={(O_1^{⊺},A_1,O_2^{⊺})}^{⊺}$; thus, the history at stage j consists of the information available prior to the assignment of the j-th treatment. The data available to estimate an optimal DTR consists of a random sample of n independent and identically distributed trajectories. For simplicity, assume that there are only two possible treatments at each stage, A_j ∈ {−1, 1} and that they are randomized, conditional on history, with known randomization probabilities. A DTR d = (d₁, d₂) is a pair of functions with d_j mapping the values in the support of H_j to the available treatment options {−1, 1}. In the next section we discuss methods for estimating a high-quality DTR, i.e., a DTR which could lead to a better benefit for the population compared with current alternatives, using data collected in a SMART.

3 Defining and Estimating an Optimal DTR

3.1 Value of a fixed DTR

The Value of a DTR d = (d₁, d₂) is:

$$V^d={\mathcal{E}}_{d_1,d_2}[Y]={\mathcal{E}}_{d_1,d_2}[r(O_1,A_1,O_2,A_2,O_3)],$$ (1)

where _d1,_d2 denotes expectation taken with respect to the distribution of the entire data trajectory (O₁, A₁, O₂, A₂, O₃) subject to the restrictions A₁ = d₁(O₁) and A₂ = d₂(O₁, A₁, O₂). A regime d is often said to be embedded in the study if d is actually employed to allocate treatments to a subset of subjects in the study, i.e., if the restrictions A₁ = d₁(O₁) and A₂ = d₂(O₁, A₁, O₂) are naturally satisfied for a non-null subset of subjects. There are 4 embedded DTRs in the hypothetical smoking cessation SMART described in Figure 1. One of these embedded DTRs is: “provide message with low degree of tailoring as the initial treatment (this is stage-1 rule, d₁); then at the second stage, give booster prevention message (this is stage-2 rule, d₂).”

When the regime of interest is an embedded DTR in the SMART, estimation of Value is relatively straightforward. The problem becomes more complex if d is not embedded. An example of a non-embedded regime in the smoking cessation study is: “as the initial treatment, provide the message with high degree of tailoring to subjects without any college education, and the message with low degree of tailoring to others; then at the second stage, give booster prevention message to the non-quitters at six months, and control message to others.” A non-embedded regime can be estimated from the observed data if it is feasible [56]; a regime d is feasible if there is a positive probability that a subject in the study would follow d. For a feasible regime d, one can use inverse probability weighting (IPW) to express the Value of the regime in terms of the generative model [57, 58, 59]. The IPW estimator is

$$V^d={\mathcal{E}}_{d_1,d_2}[Y]=\mathcal{E}\left[\frac{{\mathcal{I}}_{d_1(H_1)=A_1}{\mathcal{I}}_{d_2(H_2)=A_2}}{{\pi }_1(A_1|H_1){\pi }_2(A_2|H_2)}Y\right],$$ (2)

where denotes expectation with respect to the joint trajectory distribution, _condition denotes an indicator function taking the value 1 when the ‘condition’ holds and 0 otherwise, and π_j(a_j|h_j) = P(A_j = a_j|H_j = h_j) is the treatment allocation probability at the jth stage, j = 1, 2; these are known by design in a SMART, but must be estimated in observational studies. A plug-in estimator of the Value is

$${\widehat{V}}^d=\frac{1}{n}\sum _{i=1}^n\left[\frac{{\mathcal{I}}_{d_1(H_{1i})=A_{1i}}{\mathcal{I}}_{d_2(H_{2i})=A_{2i}}}{{\pi }_1(A_{1i}|H_{1i}){\pi }_2(A_{2i}|H_{2i})}{Y}_i\right],$$ (3)

which can be highly variable if the weights in the denominator of the expression are close to zero. Because V̂^d is a plug-in estimator there is potential for upward bias when the same data are used to construct and evaluate a DTR; an alternative approach would be to use cross-validation, however, for simplicity we do not consider this further.

3.2 Estimation of an Optimal DTR via Q-learning

There exists a variety of methods for estimating an optimal DTR, i.e., the DTR d that leads to the highest Value. Here we focus on a simple method called Q-learning [38, 60]. From dynamic programming, it is known that if Q₂(h₂, a₂) = (Y|H₂ = h₂, A₂ = a₂) and Q₁(h₁, a₁) = (max_a2 Q₂(H₂, a₂)|H₁ = h₁, A₁ = a₁), then the optimal regime satisfies $d_j^{\text{opt}}(h_j)={argmax}_{a_j}{Q}_j(h_j,a_j)$. Q-learning mimics the dynamic programming solution by estimating the Q-functions, Q_j, j = 1, 2, based on the observed data, often assuming linear working models of the form $Q_j(h_j,a_j;{\beta }_j,{\psi }_j)={\beta }_j^{⊺}{h}_{j0}+({\psi }_j^{⊺}{h}_{j1})a_j$, where h_j0 and h_j1 are (possibly different) features of h_j. A version of the Q-learning algorithm is:

1. Stage 2 regression: $({\widehat{\beta }}_2,{\widehat{\psi }}_2)=arg{min}_{{\beta }_2,{\psi }_2}{{\sum }_{i=1}^n\left(Y_i-Q_2(H_{2i},A_{2i};{\beta }_2,{\psi }_2)\right)}^2$.

2. Stage 1 dependent variable: ${\widehat{Y}}_{1i}={max}_{a_2}{Q}_2(H_{2i},a_2;{\widehat{\beta }}_2,{\widehat{\psi }}_2)$, i = 1, …, n.

3. Stage 1 regression: $({\widehat{\beta }}_1,{\widehat{\psi }}_1)=arg{min}_{{\beta }_1,{\psi }_1}{{\sum }_{i=1}^n\left({\widehat{Y}}_{1i}-Q_1(H_{1i},A_{1i};{\beta }_1,{\psi }_1)\right)}^2$.

Note that in step 2 above, the quantity Ŷ_1i is a predictor of the unobserved random variable max_a2 Q₂(H_2i, a₂), i = 1, …, n. The estimated optimal DTR using Q-learning is given by (d̂₁, d̂₂), where the stage-j optimal rule is specified as d̂_j(h_j) = arg max_aj Q_j(h_j, a_j; β̂_j, ψ̂_j), j = 1, 2. Q-learning with linear regression for two stages has been implemented in the R package qLearn [61].

3.3 Value of an Estimated DTR

For an estimated (optimal) DTR, say d̂, a fundamental question is whether or not d̂ has a higher Value than standard care. We assume that the Value of standard care is known though this is not essential. One way to compare d̂ with standard care is to construct a confidence interval for the Value of d̂, and check if the known Value of standard care lies within that interval. This is analogous to constructing a confidence interval for the unknown mean μ of a random variable and then checking if it contains a postulated value μ₀. Using (2), the Value of d̂ is given by

$$V^{\widehat{d}}={\mathcal{E}}_{{\widehat{d}}_1,{\widehat{d}}_2}Y=\mathcal{E}\left[\frac{{\mathcal{I}}_{{\widehat{d}}_1(H_1)=A_1}{\mathcal{I}}_{{\widehat{d}}_2(H_2)=A_2}}{{\pi }_1(A_1|H_1){\pi }_2(A_2|H_2)}Y\right],$$

where is taken with respect to the joint trajectory distribution but not the data used to construct d̂. Thus, V^d̂ depends both on the unknown generative distribution and on the data used to construct d̂ and is therefore a data-dependent parameter [43, 62]. Data-dependent parameters, though somewhat unusual, are appropriate when studying the performance of an estimated predictive model or decision rule since the primary focus is the performance of the estimated predictive model (conditioned on the observed data) rather than the average performance of the estimation procedure averaged across data sets [e.g., 43, 63, 64, 65]. The indicator functions present in the expression of V^d̂ make V^d̂ a non-smooth function of the data. Recall that A_j is coded to take values in {−1, 1}. For the purpose of illustration, we consider linear decision rules of the form ${\widehat{d}}_j(h_j)=\text{sign}({\widehat{\psi }}_j^{⊺}{h}_{j1})$, where h_j1 is a feature constructed from h_j; Q-learning with linear models yields decision rules of this form. In this case, the Value can be shown to equal

$$V^{\widehat{d}}=\mathcal{E}\left[\frac{Y}{{\pi }_1(A_1|H_1){\pi }_2(A_2|H_2)}{\mathcal{I}}_{min(A_1{\widehat{\psi }}_1^{⊺}{H}_{11},A_2{\widehat{\psi }}_2^{⊺}{H}_{21})\ge 0}\right],$$ (4)

which can be viewed as a weighted misclassification error with weights Y/π₁(A₁|H₁)π₂(A₂|H₂) and ‘margin’ min $(A_1{\widehat{\psi }}_1^{⊺}{H}_{11},A_2{\widehat{\psi }}_2^{⊺}{H}_{21})$ [42, 66]. The misclassification error is a well-known example of a non-regular data-dependent parameter due to the non-smoothness of the indicator function at zero. A consequence is that standard methods for inference, such as the usual n-out-of-n bootstrap [67], cannot be applied without modification (see [43] for details). In the next section we propose a simple subsampling based approach to construct valid confidence intervals for V^d̂.

4 Subsampling Confidence Interval for V^d̂

Let ${\widehat{V}}^{\widehat{d}}=n^{-1}{\sum }_{i=1}^n{Y}_i{\mathcal{I}}_{min(A_{1i}{\widehat{\psi }}_1^{⊺}{H}_{11i},A_{2i}{\widehat{\psi }}_2^{⊺}{H}_{21i})\ge 0}/\{{\pi }_1(A_{1i}|H_{1i}){\pi }_2(A_{2i}|H_{2i})\}$ denote the plug-in estimator of V^d̂. To construct a confidence interval we approximate the percentiles of the limiting distribution of $\sqrt{n}({\widehat{V}}^{\widehat{d}}-V^{\widehat{d}})$. As mentioned previously, normal approximations or standard bootstrap estimates of this limiting distribution are not consistent [43, 68]. One approach to consistently estimate the distribution of $\sqrt{n}({\widehat{V}}^{\widehat{d}}-V^{\widehat{d}})$ is to use a subsampling procedure called the m-out-of-n bootstrap [69, 70, 71]. The m-out-of-n bootstrap mimics the usual nonparametric bootstrap except that the resample size, typically denoted by m, is allowed to be data-dependent but must satisfy m →_p ∞ and m/n →_p k ∈ [0, 1] as n → ∞. Given the resample size m, we can form a (1 − η) × 100% confidence interval for V^d̂ as follows. We draw B m-out-of-n bootstrap samples and calculate the bootstrap estimates V^{^d^},^b, b = 1, …, B, and find l̂ and û, the (η/2) × 100 and (1 − η/2) × 100 percentiles of $\sqrt{m}({\widehat{V}}^{\widehat{d},b}-{\widehat{V}}^{\widehat{b}})$ respectively. The confidence set is then given by $({\widehat{V}}^{\widehat{d}}-\widehat{u}/\sqrt{m},{\widehat{V}}^{\widehat{d}}-\widehat{l}/\sqrt{m})$. Choosing the resample size m so that k = 0 remedies bootstrap inconsistency for a wide class of nonsmooth estimators [72] and in many cases the coverage of m-out-of-n bootstrap intervals increases as m decreases; however, choosing m too small will reduce efficiency [73]. Thus, m is a potentially important tuning parameter. Here we consider an adaptive choice of m proposed by Chakraborty et al. [11] for use with the Value. Details for choosing m are given in the Appendix.

5 Simulation Study

5.1 Simulation Design

In this section, we present a primary simulation study and a secondary simulation study to provide an empirical evaluation of the proposed confidence intervals. The primary study generates data from a SMART having two stages of treatment and two treatment options per stage irrespective of any intermediate measure of “treatment response” (as in the hypothetical smoking cessation SMART). The secondary study generates data from a more complicated SMART design, wherein only a subgroup of subjects (e.g. non-responders to the initial treatment) are re-randomized at the second stage; see Lei et al. [1] for examples of such studies. The purpose of the secondary study is to illustrate the wider scope of the proposed confidence intervals beyond the specific data structure offered by the motivating smoking cessation SMART.

Generative Model of the Primary Simulation Study

Here we consider a family of generative models generically described as:

$$\begin{array}{lll}{O}_1\hfill & \sim \hfill & N_3(0,I);O_2=O_1+Z,\text{where}Z\sim N_3(0,I);\hfill \\ A_j\hfill & \in \hfill & \{-1,1\},\text{with}P(A_j=1)=\frac{1}{2}\text{for}j=1,2;\hfill \\ Y\hfill & =\hfill & O_2^{⊺}{\gamma }_1+A_2{O}_2^{⊺}{\gamma }_2+e,\text{where}{\gamma }_1={\gamma }_2={(c,c,c)}^{⊺}\text{and}e\sim N(0,1).\hfill \end{array}$$

The family of models is indexed by a parameter c that represents the effect size and is varied in the set {0.5, 1, 1.5, …, 5}, resulting in 10 example scenarios. The baseline and intermediate set of covariates, denoted by O₁ and O₂ respectively, are three-component vectors; while O₁ is generated from a multivariate normal distribution with zero mean and an identity dispersion matrix I, O₂ is obtained by adding a similar multivariate normal vector Z to O₁. Following the variables that were actually collected in the smoking cessation study of Strecher et al. [54], the three variables in the above generative model can be potentially conceptualized as standardized versions of continuous scores denoting subjects' motivation to quit smoking, self-efficacy, and pre-treatment level of addiction, measured at baseline and six months post-randomization, respectively.

Generative Model of the Secondary Simulation Study

Here the generative model is described as:

$$\begin{array}{lll}{O}_1\hfill & \sim \hfill & N_3(0,I);O_2=(1+0.05A_1)O_1+0.5Z,\text{where}Z\sim N_3(0,I);\hfill \\ A_j\hfill & \in \hfill & \{-1,1\},\text{with}P(A_j=1)=\frac{1}{2}\text{for}j=1,2;\hfill \\ R\hfill & =\hfill & {\mathcal{I}}_{\{{\sum }_{k=1}^3{O}_{2k}\le -1\}};\hfill \\ Y_1\hfill & =\hfill & O_1^{⊺}{\gamma }_1+A_1{O}_1^{⊺}{\gamma }_2+e_1;\hfill \\ Y_2\hfill & =\hfill & O_2^{⊺}{\gamma }_1+A_2{O}_2^{⊺}{\gamma }_2+e_2,\text{where}{\gamma }_1={\gamma }_2={(c,c,c)}^{⊺}\text{and}{e}_1,e_2\stackrel{i.i.d}{\sim }N(0,1);\hfill \\ Y\hfill & =\hfill & R\cdot Y_1+(1-R)\cdot Y_2.\hfill \end{array}$$

Here R is an indicator of treatment response at the end of stage 1; also, Y₁ and Y₂ are the stage-specific outcomes combined to construct the final outcome Y. The treatment at stage 2 is given only to the non-responders of stage 1; responders do not receive any new treatment. The rest of the variables can be conceptualized as in the primary simulation study. The effect size parameter c is set as 0.5.

Q-learning is employed to estimate the optimal DTR. The correctly specified working models for the Q-functions are given by $Q_j(H_j,A_j;{\beta }_j,{\psi }_j)={\beta }_j^{⊺}{H}_{j0}+{\psi }_j^{⊺}{H}_{j1}{A}_j$, j = 1, 2, where $H_{20}={(1,O_1^{⊺},A_1{O}_1^{⊺}{A}_1)}^{⊺}$, $H_{21}={(1,O_2^{⊺},A_1)}^{⊺}$, and $H_{10}=H_{11}={(1+O_1^{⊺})}^{⊺}$. Given the observed data, and thus an estimated DTR d̂, the true Value of d̂, i.e. V^d̂, is approximated using a separate Monte Carlo evaluation data set of size 10000. We compare the standard percentile bootstrap with the proposed m-out-of-n bootstrap in terms of the mean coverage and mean width of nominal 95% CIs. Comparisons are based on 1000 simulated data sets, each of size n = 200, and 1000 bootstrap replications.

5.2 Results

Results of the primary simulation study are shown in the top part of Table 1. The standard bootstrap (n-out-of-n) shows the problem of under-coverage in all the 10 examples tried, each example denoting a different effect size. The extent of under-coverage is often severe, the lowest observed coverage rate for the nominal 95% bootstrap confidence interval is 83.8%. On the other hand, the proposed subsampling (m-out-of-n, with m chosen via double bootstrap) confidence intervals provide nominal coverage rate in almost all examples. The only exception occurs when the effect size is the smallest in the range we considered (c = 0.5); in this example, the coverage of the sub-sampling confidence interval (93.4%) falls marginally below the nominal rate using a binomial test of proportion, while still offering considerable improvement over the standard bootstrap (86.6%). Results of the secondary simulation study are shown in the lower part of Table 1. Here also the subsampling approach offers improved coverage (94.2%) over the standard bootstrap (92.0%). As expected, in both primary and secondary simulation studies, confidence intervals constructed via subsampling are wider than those constructed using the standard bootstrap.

Table 1.

Coverage rate and mean width of nominal 95% CIs for V_d̂ based on 1000 Monte Carlo simulations (n = 200). Coverage rates significantly smaller than the nominal rate are written in bold font.

	Coverage		Mean Width
c	Bootstrap	Subsampling	Bootstrap	Subsampling
Primary Simulation Study
0.5	0.866	0.934	0.976	1.197
1	0.894	0.966	1.616	1.968
1.5	0.838	0.948	2.324	2.829
2	0.882	0.952	3.036	3.771
2.5	0.894	0.956	3.791	4.642
3	0.876	0.954	4.508	5.481
3.5	0.838	0.944	5.323	6.616
4	0.858	0.944	5.898	7.188
4.5	0.854	0.942	6.686	8.138
5	0.870	0.940	7.421	9.063
Secondary Simulation Study
0.5	0.920	0.942	0.704	0.730

6 Discussion

We proposed a subsampling-based confidence interval for the Value of an estimated optimal DTR when estimation is done using Q-learning. The proposed method is adaptive in that it uses a data-driven resample size. The method is conceptually simple, easily implemented without specialized software, and self-tuning via the double bootstrap. In simulated experiments the proposed method delivered improved performance over the standard bootstrap.

While we used the IPW estimator of the Value in our setup, an alternative is to use the Augmented IPW (AIPW) estimator which is more complex but generally more efficient [e.g., 42]. Whether and how the proposed subsampling confidence interval can be applied in conjunction with the AIPW estimator remains an open question, and thus can be an interesting topic for future research.

An important application of the proposed method is the comparison of two competing DTRs. We discussed the case where the Value of a competing regime (say, standard care) was known and compared with an estimated optimal regime. However, the proposed method can also be applied to construct a confidence interval for the difference between the Value of the estimated optimal regime and that of a competing fixed regime. In particular, one can bootstrap the difference between V^{^d̂} and the estimator given in (3) to form a confidence interval for the difference in Values.

Appendix

Data-driven m-out-of-n subsampling

From (4) it can be seen that the stability of V^d̂ across SMART data sets will depend on the proportion of subjects for which min $(A_1{\widehat{\psi }}_1^{⊺}{H}_{11},A_2{\widehat{\psi }}_2^{⊺}{H}_{21})\approx 0$. Furthermore, it is important to note that in a SMART (as in any randomized study), treatment effect sizes are often small, in accordance with the principle of equipoise [74]. This prevalence of small effect sizes implies that the proportion of subjects for which min $(A_1{\widehat{\psi }}_1^{⊺}{H}_{11},A_2{\widehat{\psi }}_2^{⊺}{H}_{21})\approx 0$ is non-negligible. Intuitively, the estimator will ‘jitter’ more across data sets as this proportion increases. Thus, a natural approach is to let the resample size m depend on an estimator of this proportion.

Let $B_{\mathit{\text{ji}}}=(H_{j0,i}^{⊺},H_{j1,i}^{⊺}{A}_{\mathit{\text{ji}}})$ denote the i-th row of the design matrix for the stage-j regression in Q-learning, and θ_j* be the true value of ${\theta }_j={({\beta }_j^{⊺},{\psi }_j^{⊺})}^{⊺}$, for j = 1, 2. Then ${\widehat{\sum }}_j^{\ast }$, the plug-in estimator of the asymptotic covariance of θ̂_j is given by:

$$\begin{array}{lll}{\widehat{\sum }}_2^{\ast }\hfill & =\hfill & n{\left(\sum _{i=1}^n{B}_{2i}{B}_{2i}^{⊺}\right)}^{-1}{\sum _{i=1}^n{B}_{2i}{B}_{2i}^{⊺}{\left(Y_i-B_{2i}^{⊺}{\theta }_2^{\ast }\right)}^2\left(\sum _{i=1}^n{B}_{2i}{B}_{2i}^{⊺}\right)}^{-1},\hfill \\ {\widehat{\sum }}_1^{\ast }\hfill & =\hfill & n{\left(\sum _{i=1}^n{B}_{1i}{B}_{1i}^{⊺}\right)}^{-1}\sum _{i=1}^n{B}_{1i}{B}_{1i}^{⊺}{\left({\widehat{Y}}_{1i}-B_{1i}^{⊺}{\theta }_1^{\ast }\right)}^2{\left(\sum _{i=1}^n{B}_{1i}{B}_{1i}^{⊺}\right)}^{-1}.\hfill \end{array}$$

Finally, let Σ̂_j denote the sub-matrix of ${\widehat{\sum }}_j^{\ast }$ corresponding to elements of ψ_j; thus Σ̂_j represent the asymptotic covariance of ψ̂_j. Furthermore, let ${\chi }_{1,1-\nu }^2$ denote the (1 − ν) × 100 percentile of a Chi-Square distribution with one degree of freedom. Define ${\psi }_j^{\ast }$, j = 1, 2 to be the almost sure limiting values of ψ̂_j, j = 1, 2 (see [68] for conditions under which these limits exist). Define

$$\widehat{p}=\underset{\nu \in (0,1)}{inf}\left[\frac{1}{n}\sum _{i=1}^n\mathcal{I}\left\{min\left(\frac{n{(H_{11}^{⊺}{\widehat{\psi }}_1)}^2}{H_{11}^{⊺}{\widehat{\sum }}_1{H}_{11}},\frac{n{(H_{21}^{⊺}{\widehat{\psi }}_2)}^2}{H_{21}^{⊺}{\widehat{\sum }}_2{H}_{21}}\right)\le {\chi }_{1,1-\nu }^2\right\}+\nu \right],$$

where {·} is an indicator function. Then, under mild regularity conditions, p̂ is a conservative estimator of $p=P\{min(A_1{\psi }_1^{\ast ⊺}{H}_{11},A_2{\psi }_2^{\ast ⊺}{H}_{21})=0\}$ in the sense that for any fixed value ε > 0, P(p̂ ≥ p = ε) → 1; furthermore, when p = 0 it follows that p̂ →_p p. We define the class of data-driven resample sizes, indexed by a tuning parameter α > 0, as m̂_α = n^{(1+α(1−p̂))/(1+α)}. Note that m̂₀ ≡ n and m̂_∞ = n^1−p̂; see [11] for further discussion of this class. It can be shown that for 0 < ℓ < L < ∞, sup_ℓ≤_α≤_L m̂_α = o_p(n) if p > 0, and inf_ℓ≤_α≤_L m̂_α/n →_p 1 if p = 0 (see [11] for a proof of a similar result). For this method to work, a value of the tuning parameter α must be chosen. A data-driven approach to choosing α can be devised using the double bootstrap; such an algorithm is described below. R code implementing the proposed method is freely available from the authors.

Double bootstrap algorithm for choosing α

To form a (1 − η) × 100% confidence interval (CI) for V^d̂, we provide a double bootstrap procedure [see 75, for details] for choosing the tuning parameter α.

Suppose we have estimated ψ̂_j, j = 1, 2, subsequently d̂ and V^{^d̂} from the original data. Consider a grid of candidate values for α; we used {0.025, 0.05, 0.075, …, 1}. The algorithm is as follows.

1. Draw B₁ n-out-of-n first-stage bootstrap samples from the data and calculate the bootstrap estimates V^{^d^,(b1)}, b₁ = 1, …, B₁. Fix α at the smallest value in the grid.

2. Compute the corresponding values of ${\widehat{m}}_{\alpha }^{(b_1)}=n^{(1+\alpha (1-{\widehat{p}}^{(b_1)}))/(1+\alpha )}$, b₁ = 1, …,B₁.

3. Conditional on each first-stage bootstrap sample, draw B₂ m̂^(b₁)-out-of-n second-stage (nested) bootstrap samples and calculate the double bootstrap versions of the estimate V^{^d̂},(^b1b2), b₁ = 1, …, B₁, b₂ = 1, …, B₂.

4. For b₁ = 1, …, B₁, compute the (η/2) × 100 and (1 − η/2) × 100 percentiles of $\left\{\sqrt{{\widehat{m}}^{(b_1)}}\left({\widehat{V}}^{\widehat{b},(b_1{b}_2)}-{\widehat{V}}^{\widehat{d},(b_1)}\right),b_2=1,\dots ,B_2\right\}$; say ${\widehat{l}}_{\text{DB}}^{(b_1)}$ and ${\widehat{u}}_{\text{DB}}^{(b_1)}$ respectively. Construct the double centered percentile bootstrap [67] CI from the b₁-th first-stage bootstrap data as $({\widehat{V}}^{\widehat{d},(b_1)}-{\widehat{u}}_{\text{DB}}^{(b_1)}/\sqrt{{\widehat{m}}^{(b_1)}},{\widehat{V}}^{\widehat{d},(b_1)}-{\widehat{l}}_{\text{DB}}^{(b_1)}/\sqrt{{\widehat{m}}^{(b_1)}})$, b₁ = 1, …, B₁.

5. Estimate the coverage rate of the double bootstrap CI from all the first-stage bootstrap data sets as

   $$\frac{1}{B_1}\sum _{b_1=1}^{B_1}\mathcal{I}\left\{{\widehat{V}}^{\widehat{d},(b_1)}-{\widehat{u}}_{\text{DB}}^{(b_1)}/\sqrt{{\widehat{m}}^{(b_1)}}\le {\widehat{V}}^{\widehat{d}}\le {\widehat{V}}^{\widehat{d},(b_1)}-{\widehat{l}}_{\text{DB}}^{(b_1)}/\sqrt{{\widehat{m}}^{(b_1)}}\right\}.$$

6. If the above coverage rate is at or above the nominal level, then pick the current value of α as the final value. Otherwise, increment α to the next highest value in the grid.

7. Repeat steps 2 – 6, until the coverage rate of the double bootstrap CI, attains the nominal coverage rate, or the grid is exhausted.