Targeted Learning of the Mean Outcome under an Optimal Dynamic Treatment Rule

Abstract

We consider estimation of and inference for the mean outcome under the optimal dynamic two time-point treatment rule defined as the rule that maximizes the mean outcome under the dynamic treatment, where the candidate rules are restricted to depend only on a user-supplied subset of the baseline and intermediate covariates. This estimation problem is addressed in a statistical model for the data distribution that is nonparametric beyond possible knowledge about the treatment and censoring mechanism. This contrasts from the current literature that relies on parametric assumptions. We establish that the mean of the counterfactual outcome under the optimal dynamic treatment is a pathwise differentiable parameter under conditions, and develop a targeted minimum loss-based estimator (TMLE) of this target parameter. We establish asymptotic linearity and statistical inference for this estimator under specified conditions. In a sequentially randomized trial the statistical inference relies upon a second-order difference between the estimator of the optimal dynamic treatment and the optimal dynamic treatment to be asymptotically negligible, which may be a problematic condition when the rule is based on multivariate time-dependent covariates. To avoid this condition, we also develop TMLEs and statistical inference for data adaptive target parameters that are defined in terms of the mean outcome under the estimate of the optimal dynamic treatment. In particular, we develop a novel cross-validated TMLE approach that provides asymptotic inference under minimal conditions, avoiding the need for any empirical process conditions. We offer simulation results to support our theoretical findings.

1 Introduction

Suppose we observe n in4dependent and identically distributed observations of a time-dependent random variable consisting of baseline covariates, initial treatment and censoring indicator, intermediate covariates, subsequent treatment and censoring indicator, and a final outcome. For example, this could be data generated by a sequentially randomized controlled trial (RCT) in which one follows up a group of subjects, and treatment assignment at two time points is sequentially randomized, where the probability of receiving treatment might be determined by a baseline covariate for the first-line treatment, and time-dependent intermediate covariate (such as a biomarker of interest) for the second-line treatment [1]. Such trials are often called sequential multiple assignment randomized trials (SMART). A dynamic treatment rule deterministically assigns treatment as a function of the available history. If treatment is assigned at two time points, then this dynamic treatment rule consists of two rules, one for each time point [1–4]. The mean outcome under a dynamic treatment is a counterfactual quantity of interest representing what the mean outcome would have been if everybody would have received treatment according to the dynamic treatment rule [5–11]. Dynamic treatments represent prespecified multiple time-point interventions that at each treatment-decision stage are allowed to respond to the currently available treatment and covariate history. Examples of multiple time-point dynamic treatment regimes are given in Lavori and Dawson [12, 13]; Murphy [14]; Rosthøj et al. [15]; Thall et al. [16, 17]; Wagner et al. [18]; Petersen et al. [19]; van der Laan and Petersen [20]; and Robins et al. [21], ranging from rules that change the dose of a drug, change or augment the treatment, to making a decision on when to start a new treatment, in response to the history of the subject.

More recently, SMART designs have been implemented in practice: Lavori and Dawson [12, 22]; Murphy [14]; Thall et al. [16]; Chakraborty et al. [23]; Kasari [24]; Lei et al. [25]; Nahum-Shani et al. [26, 27]; Jones [28]; Lei et al. [25]. For an extensive list of SMARTs, we refer the reader to the website http://methodology.psu.edu/ra/adap-inter/projects. For an excellent and recent overview of the literature on dynamic treatments we refer to Chakraborty and Murphy [29].

We define the optimal dynamic multiple time-point treatment regime as the rule that maximizes the mean outcome under the dynamic treatment, where the candidate rules are restricted to only respond to a user-supplied subset of the baseline and intermediate covariates. The literature on Q-learning shows that we can describe the optimal dynamic treatment among all dynamic treatments in a sequential manner [14, 30–33]. The optimal rule can be learned through fitting the likelihood and then calculating the optimal rule under this fit of the likelihood. This approach can be implemented with maximum likelihood estimation based on parametric models. It has been noted (e.g., Robins [32], Chakraborty and Murphy [29]) that the estimator of the parameters of one of the regressions (except the first one) when using parametric regression models is a non-smooth function of the estimator of the parameters of the previous regression, and that this results in non-regularity of the estimators of the parameter vector. This raises challenges for obtaining statistical inference, even when assuming that these parametric regression models are correctly specified. Chakraborty and Murphy [29] discuss various approaches and advances that aim to resolve this delicate issue such as inverting hypothesis testing [32], establishing non-normal limit distributions of the estimators (E. Laber, D. Lizotte, M. Qian, S. Murphy, submitted), or using the m out of n bootstrap.

Murphy [30] and Robins [31, 32] developed structural nested mean models tailored to optimal dynamic treatments. These models assume a parametric model for the “blip function” defined as the additive effect of a blip in current treatment on a counterfactual outcome, conditional on the observed past, in the counterfactual world in which future treatment is assigned optimally. Statistical inference for the parameters of the blip function proceeds accordingly, but Robins [32] points out the irregularity of the estimator, resulting in some serious challenges for statistical inference as referenced above. Structural nested mean models have also been generalized to blip functions that condition on a (counterfactual) subset of the past, thereby allowing the learning of optimal rules that are restricted to only using this subset of the past [32] and Section 6.5 in van der Laan and Robins [34].

An alternative approach, referenced as the direct approach in Chakraborty and Murphy [29], uses marginal structural models (MSMs) for the dynamic regime-specific mean outcome for a user-supplied class of dynamic treatments. If one assumes the marginal structural models are correctly specified, then the parameters of the marginal structural model map into a dynamic treatment that is optimal among the user-supplied class of dynamic regimes. In addition, the MSM also provides the complete dose–response curve, that is, the mean counterfactual outcome for each dynamic treatment in the user-supplied class. This generalization of the original marginal structural models for static interventions to MSMs for dynamic treatments was developed independently by Orellana et al. [35]; van der Laan and Petersen [20]. These articles present inverse probability of treatment and censoring weighted (IPCW) estimators and double robust augmented IPCW estimators based on general longitudinal data structures, allowing for right censoring, time-dependent covariates, and survival outcomes. Double robust estimating equation-based methods that estimate the nuisance parameters with sequential parametric regression models using clever covariates were developed for static intervention MSMs by Bang and Robins [36]. An analogous targeted minimum loss-based estimator (TMLE) [37–39] was developed for marginal structural models for a user-supplied class of dynamic treatments by Petersen et al. [40]. This estimator builds on the TMLE for the mean outcome for a single dynamic treatment developed by van der Laan and Gruber [41]. Additional application papers of interest are [42–44] which involve fitting MSMs for dynamic treatments defined by treatment-tailoring threshold using IPCW methods.

Each of the above referenced approaches for learning an optimal dynamic treatment that also aims to provide statistical inference relies on parametric assumptions: obviously, Q-learning based on parametric models, but also the structural nested mean models and the marginal structural models both rely on parametric models for the blip function and dose–response curve, respectively. As a consequence, even in a SMART, the statistical inference for the optimal dynamic treatment heavily relies on assumptions that are generally believed to be false, and will thus be expected to be biased.

To avoid such biases, we define the statistical model for the data distribution as nonparametric, beyond possible knowledge about the treatment mechanism (e.g., known in an RCT) and censoring mechanism. This forces us to define the optimal dynamic treatment and the corresponding mean outcome as parameters defined on this nonparametric model, and to develop data adaptive estimators of the optimal dynamic treatment. In order to not only consider the most ambitious fully optimal rule, we define the V-optimal rules as the optimal rule that only uses a user-supplied subset V of the available covariates. This allows us to consider suboptimal rules that are easier to estimate and thereby allow for statistical inference for the counterfactual mean outcome under the suboptimal rule. This is analogous to the generalized structural nested mean models whose blip functions only condition on a counterfactual subset of the past. In a companion article we describe how to estimate the V-optimal rule.

In Example 4 of Robins et al. [45], the authors develop an asymptotic confidence set for the optimal treatment regime in an RCT under a large semiparametric model that only assumes that the treatment mechanism is known. This confidence set is certainly of interest and warrants further consideration in the optimal treatment literature. They get this confidence set by deriving the efficient influence curve for the mean squared blip function. They propose selecting a data adaptive estimate of the optimal treatment rule by a particular cross-validation scheme over a set of basis functions, and show that this estimator achieves a data adaptive rate of convergence under smoothness assumptions on the blip function. Our work is distinct from this earlier work in that the earlier work does not directly consider the mean outcome under the optimal rule and only considers data generated by a point treatment RCT.

In this article we describe how to obtain semiparametric inference about the mean outcome under the two time point V-optimal rule. We will show that the mean outcome under the optimal rule is a pathwise differentiable parameter of the data distribution, indicating that it is possible to develop asymptotically linear estimators of this target parameter under conditions. In fact, we obtain the surprising result that the pathwise derivative of this target parameter equals the pathwise derivative of the mean counterfactual outcome under a given dynamic treatment rule set at the optimal rule, treating the latter as known. By a reference to the current literature for double robust and efficient estimation of the mean outcome under a given rule, we then obtain a TMLE for the mean outcome under the optimal rule. Subsequently, we prove asymptotic linearity and efficiency of this TMLE, allowing us to construct confidence intervals for the mean outcome under the optimal dynamic treatment or its contrast with respect to a standard treatment. Thus, contrary to the irregularity of the estimators of the unknown parameters in the semiparametric structural nested mean model, we can construct regular estimators of the mean outcome under the optimal rule in the nonparametric model.

In a SMART the statistical inference would only rely upon a second-order difference between the estimator of the optimal dynamic treatment and the optimal dynamic treatment itself to be asymptotically negligible. This is a reasonable condition if we restrict ourselves to rules only responding to a one-dimensional time-dependent covariate, or if we are willing to make smoothness assumptions. To avoid this condition, we also develop TMLEs and statistical inference for data adaptive target parameters that are defined in terms of the mean outcome under the estimate of the optimal dynamic treatment (see van der Laan et al. [46] for a general approach for statistical inference for data adaptive target parameters). In particular, we develop a novel cross-validated TMLE (CV-TMLE) approach that provides asymptotic inference under minimal conditions.

For the sake of presentation, we focus on two time point treatments in this article. In the appendices of our earlier technical reports [47, 48] we generalize these results to general multiple time point treatments, and develop general (sequential) super-learning based on the efficient CV-TMLE of the risk of a candidate estimator. In this appendix we also develop a TMLE of a projection of the blip functions on a parametric working model (with corresponding statistical inference, which presents a result of interest in its own right). We emphasize that this technical report is distinct from our companion paper in this issue, which focuses on the data adaptive estimation of optimal treatment strategies.

1.1 Organization of article

Section 2 defines the mean outcome under the optimal rule as a causal parameter and gives identifiability assumptions under which the causal parameter is identified with a statistical parameter of the observed data distribution.

The remainder of the paper describes strategies to estimate the counterfactual mean outcome under the optimal rule and related quantities. This paper assumes that we have an estimate of the optimal rule in our semiparametric model. In our companion paper we describe how to obtain estimates of the V-optimal rule.

The first part of this article concerns estimation of the mean outcome under the optimal rule. Section 3 establishes the pathwise differentiability of the mean outcome under the V-optimal rule conditions. A closed form expression for the efficient influence curve for this statistical parameter is given, which represents a key ingredient in semiparametric inference for the statistical target parameter. We obtain the surprising result that, under straightforward conditions, estimating the mean outcome under the unknown optimal treatment rule is the same in first order as estimating the mean outcome under the optimal rule when the rule is known from the outset. Section 4 presents the key properties of a TMLE for the mean outcome under the optimal rule, which is presented in detail in “TMLE of the mean outcome under a given rule” in Appendix B due to its similarity to TMLEs presented previously in the literature. Section 5 presents an asymptotic linearity theorem for this TMLE and corresponding statistical inference.

The second part of this article concerns statistical inference for data adaptive target parameters that are defined in terms of the mean outcome under the estimate of the optimal dynamic treatment, thereby avoiding the consistency and rate condition for the fitted V-optimal rule as required for asymptotic linearity of the TMLE of the mean outcome under the actual V-optimal rule. These results are of interest in practice because an estimated, possibly suboptimal, rule will be implemented in the population, not some unknown optimal rule. Section 6 presents an asymptotic linearity theorem for the TMLE presented in Section 4, but now with the target parameter defined as the mean outcome under the estimated rule. In Section 7 we present the CV-TMLE framework. A specific CV-TMLE algorithm is described in “CV-TMLE of the mean outcome under data adaptive V-optimal rule” in Appendix B due to its similarity to CV-TMLEs presented previously in the literature. The CV-TMLE provides asymptotic inference under minimal conditions for the mean outcome under a dynamic treatment fitted on a training sample, averaged across the different splits in training sample and validation sample. Both results allow us to construct confidence intervals that have the correct asymptotic coverage of the random true target parameter, and the fixed mean outcome under the optimal rule under conditions, but statistical inference based on the CV-TMLE does not require an empirical process condition that would put a brake on the allowed data adaptivity of the estimator.

Section 8 presents the simulation methods. The simulations estimate the optimal rule using an ensemble algorithm presented in our companion paper, and then given this estimate apply the estimators of the optimal rule presented in this paper. Section 9 presents the coverage and efficiency of the various estimators in our simulation. Appendix C gives analytic intuition as to why some of the simulation results may have occurred. Section 10 closes with a discussion and directions for future work.

All proofs can be found in Appendix A.

2 Formulation of optimal dynamic treatment estimation problem

Suppose we observe n i.i.d. copies $O_1,\dots ,O_n\in \mathcal{O}$ of

$$O=\left(L\right(0),A(0),L(1),A(1),Y)\sim P_0,$$

where A(j) = (A₁(j), A₂(j)), A₁(j) is a binary treatment, and A₂(j) is an indicator of not being right censored at “time” j, j = 0, 1. That is, A₂(0) = 0 implies that (L(1), A₁(1), Y) is n ot observed, and A₂(1) = 0 implies that Y is not observed. Each time point j has covariates L(j) that precede treatment, j = 0, 1, and the outcome of interest is given by Y and occurs after time point 1. For a time-dependent process X(·), we use the notation $\stackrel{‒}{X}\left(t\right)=\left(X\right(s):s\le t)$, where $\stackrel{‒}{X}(-1)=\varnothing$. Let $\mathcal{M}$ be a statistical model that makes no assumptions on the marginal distribution Q_0,L(0) of L(0) and the conditional distribution Q_0,L(1) of L(1), given A(0), L(0), but might make assumptions on the conditional distributions g_0A(j) of A(j), given $\stackrel{‒}{A}(j-1)$, $\stackrel{‒}{L}\left(j\right)$, j = 0, 1. We will refer to g₀ as the intervention mechanism, which can be factorized in a treatment mechanism g₀₁ and censoring mechanism g₀₂ as follows:

$$g_0\left(O\right)=\prod _{j=1}^2{g}_{01}\left(A_1\right(j)\mid \stackrel{‒}{A}(j-1),\stackrel{‒}{L}(j\left)\right)g_{02}\left(A_2\right(j)\mid A_1(j),\stackrel{‒}{A}(j-1),\stackrel{‒}{L}(j\left)\right).$$

In particular, the data might have been generated by a SMART, in which case g₀₁ is known.

Let V(1) be a function of (L(0), A(0), L(1)), and let V(0) be a function of L(0). Let V = (V(0), V(1)). Consider dynamic treatment rules V(0) → d_A(0)(V(0)) ∈ {0, 1} × {1} and (A(0), V(1)) → d_A(1)(A(0), V(1)) ∈ {0, 1} × {1} for assigning treatment A(0) and A(1), respectively, where the rule for A(0) is only a function of V(0), and the rule for A(1) is only a function of (A(0), V(1)). Note that these rules are restricted to set the censoring indicators A₂(j) = 1, j = 0, 1. Let $\mathcal{D}$ be the set of all such rules. We assume that V(0) is a function of V(1) (i.e., observing V(1) includes observing V(0)), but in the theorem below we indicate an alternative assumption. For $d\in \mathcal{D}$, we let

$$d\left(a\right(0),v)\equiv \left(d_{A\left(0\right)}\left(v\right(0\left)\right),d_{A\left(1\right)}\left(a\right(0),v(1\left)\right)\right).$$

If we assume a structural equation model [7] for variables stating that

$$L\left(0\right)=f_{L\left(0\right)}\left(U_{L\left(0\right)}\right)$$

$$A\left(0\right)=f_{A\left(0\right)}\left(L\right(0),U_{A\left(0\right)})$$

$$L\left(1\right)=f_{L\left(1\right)}\left(L\right(0),A(0),U_{L\left(1\right)})$$

$$A\left(1\right)=f_{A\left(1\right)}\left(\stackrel{‒}{L}\right(1),A(0),U_{A\left(1\right)})$$

$$Y=f_Y\left(\stackrel{‒}{L}\right(1),\stackrel{‒}{A}(1),U_Y),$$

where the collection of functions f = (f_L(0), f_A(0), f_L(1), f_A(1)) is unspecified or partially specified, we can define counterfactuals Y_d defined by the modified system in which the equations for A(0), A(1) are replaced by A(0) = d_A(0)(V(0)) and A(1) = d_A(1)(A(0), V(1)), respectively. Denote the distribution of these counter-factual quantities as P_0,d, where we note that P_0,d is implied by the collection of functions f and the joint distribution of exogeneous variables (U_L(0), U_A(0), U_L(1), U_A(1), U_Y). We can now define the causally optimal rule under P_0,d as $d_0^{\ast }={\text{arg max}}_{d\in \mathcal{D}}{E}_{P_{0,d}}{Y}_d$. If we assume a sequential randomization assumption stating that A(0) is independent of U_L(1), U_Y, given L(0), and A(1) is independent of U_Y, given $\stackrel{‒}{L}\left(1\right)$, A(0), then we can identify P_0,d with observed data under the distribution P₀ using the G-computation formula:

$$p_{0,d}\left(L\right(0),A(0),L(1),A(1),Y)\equiv I(A=d(A\left(0\right),V\left)\right)q_{0,L\left(0\right)}\left(L\right(0\left)\right)q_{0,L\left(1\right)}\left(L\right(1)\mid L(0),A(0\left)\right)q_{0,Y}(Y\mid \stackrel{‒}{L}(1),\stackrel{‒}{A}(1\left)\right),$$ (1)

where p_0,d is the density of P_0,d and q_0,L(0), q_0,L(1), and q_0,Y are the densities for Q_0,L(0), Q_0,L(1), and Q_0,Y, respectively, where Q_0,Y represents the distribution of Y given $\stackrel{‒}{L}\left(1\right)$, $\stackrel{‒}{A}\left(1\right)$. We assume that all densities above are absolutely continuous with respect to some dominating measure μ. We have a similar identifiability result/G-computation formula under the Neyman-Rubin causal model [8]. For the right censoring indicators A₂(0) and A₂(1), we note the parallel between the coarsening at random assumption and the sequential randomization assumption [49]. Thus here we have encoded our missingness assumptions in our causal assumptions.

More generally, for a distribution $P\in \mathcal{M}$ we can define the G-computation distribution P_d as the distribution with density

$$p_d\left(L\right(0),A(0),L(1),A(1),Y)\equiv I(A=d(A\left(0\right),V\left)\right)q_{L\left(0\right)}\left(L\right(0\left)\right)q_{L\left(1\right)}\left(L\right(1)\mid L(0),A(0\left)\right)q_Y(Y\mid \stackrel{‒}{L}(1),\stackrel{‒}{A}(1\left)\right),$$

where q_L(0), q_L(1), and q_Y are the counterparts to q_0,L(0), q_0,L(1), and q_0,Y, respectively, under P.

For the remainder of this article, if for a static or dynamic intervention d, we use notation L_d (or Y_d, O_d) we mean the random variable with the probability distribution P_d in (1) so that all of our quantities are statistical parameters. For example, the quantity E_P0(Y_a(0)a(1)|V_a(0)(1)) defined in the next theorem denotes the conditional expectation of Y_a(0)a(1), given V_a(0)(1), under the probability distribution P_0,a(0)a(1) (i.e., G-computation formula presented above for the static intervention (a(0), a(1)). In addition, if we write down these parameters for some P_d, we will automatically assume the positivity assumption at P required for the G-computation formula to be well defined. For that it will suffice to assume the following positivity assumption at P:

$$\begin{array}{c}\hfill {\text{Pr}}_p\left(0<\underset{a_1\in \{0,1\}}{\min }{g}_{0A\left(0\right)}(a_1,1\mid L(0\left)\right)\right)=1\hfill \\ \hfill {\text{Pr}}_p\left(0<\underset{a_1\in \{0,1\}}{\min }{g}_{0A\left(1\right)}(a_1,1\mid \stackrel{‒}{L}(1),A(0\left)\right)\right)=1.\hfill \end{array}$$ (2)

The strong positivity assumption will be defined as the above assumption, but where the 0 is replaced by a δ > 0.

We now define a statistical parameter representing the mean outcome Y_d under P_d. For any rule $d\in \mathcal{D}$, let

$${\Psi }_d\left(P\right)\equiv E_{p_d}{Y}_d.$$

For a distribution P, define the V-optimal rule as

$$d_P=\arg \underset{d\in \mathcal{D}}{\max }{E}_{p_d}{Y}_d.$$

For simplicity, we will write d₀ instead of d_P0 for the V-optimal rule under P₀. Define the parameter mapping $\Psi :\mathcal{M}\to \text{IR as}\Psi \left(P\right)=E_{P_{d_P}}{Y}_{d_P}$. The first part of this article is concerned with inference for the parameter

$${\Psi }_0\equiv \Psi \left(P_0\right)=E_{p_{0,d_0}}{Y}_{d_0}.$$

Under our identifiability assumptions, d₀ is equal to the causally optimal rule $d_0^{\ast }$. Even if the sequential randomization assumption does not hold, the statistical parameter ψ₀ represents a statistical parameter of interest in its own right. We will not concern ourselves with the sequential randomization assumption for the remainder of this paper.

The next theorem presents an explicit form of the V-optimal individualized treatment rule d₀ as a function of P₀.

Theorem 1. Suppose V(0) is a function of V(1). The V-optimal rule d₀ can be represented as the following explicit parameter of P₀:

$$\begin{array}{cc}\hfill {\stackrel{‒}{Q}}_{20}\left(a\right(0),v(1\left)\right)& =E_{P_0}\left(Y_{a\left(0\right),A\left(1\right)=(1,1)}\mid V_{a\left(0\right)}\left(1\right)=v\left(1\right)\right)-E_{P_0}\left(Y_{a\left(0\right),A\left(1\right)=(0,1)}\mid V_{a\left(0\right)}\left(1\right)=v\left(1\right)\right)\hfill \\ \hfill d_{0,A\left(1\right)}\left(A\right(0),V(1\left)\right)& =\left(I\left({\stackrel{‒}{Q}}_{20}\right(A\left(0\right),V\left(1\right))>0),1\right)\hfill \\ \hfill {\stackrel{‒}{Q}}_{10}\left(v\right(0\left)\right)& =E_{P_0}\left(Y_{(1,1),d_{0,A\left(1\right)}}\mid V\left(0\right)\right)-E_{P_0}\left(Y_{(0,1),d_{0,A\left(1\right)}}\mid V\left(0\right)\right)\hfill \\ \hfill d_{0,A\left(0\right)}\left(V\right(0\left)\right)& =\left(I\left({\stackrel{‒}{Q}}_{10}\right(V\left(0\right))>0),1\right),\hfill \end{array}$$

where a(0) ∈ {0, 1} × {1}. If V(1) does not include V(0), but, for all (a(0), a(1)) ∈ {{0, 1} × {1}}²,

$$E_{P_0}\left(Y_{a\left(0\right),a\left(1\right)}\mid V\left(0\right),V_{a\left(0\right)}\left(1\right)\right)=E_{P_0}\left(Y_{a\left(0\right),a\left(1\right)}\mid V_{a\left(0\right)}\left(1\right)\right),$$ (3)

then the above expression for the V-optimal rule d₀ is still true.

3 The efficient influence curve of the mean outcome under V-optimal rule

In this section we establish the pathwise differentiability of Ψ and give an explicit expression for the efficient influence curve [34, 50, 51]. Before presenting this result, we give the efficient influence curve for the parameter $\Psi :\mathcal{M}\to \mathbb{R}$ where Ψ_d(P) ≡ E_PY_d and the rule d = (d_A(0), d_A(1)) $\in \mathcal{D}$ is treated as known. This influence curve has previously been presented in the literature [36, 41]. The parameter mapping Ψ_d has efficient influence curve:

$$D^{\ast }(d,P)=\sum _{k=0}^2{D}_k^{\ast }(d,P)$$

where

$$\begin{array}{cc}\hfill D_0^{\ast }(d,P)& =E_P\left[Y_d\mid L\left(0\right),A\left(0\right)=d_{A\left(0\right)}\left(V\right(0\left)\right)\right]-E_P{Y}_d\hfill \\ \hfill D_1^{\ast }(d,P)& =\frac{I\left(A\left(0\right)=d_{A\left(0\right)}\left(V\right(0\left)\right)\right)}{g_{A\left(0\right)}\left(O\right)}\times \left(E_P\left[(Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1)\right]-E_P\left[Y_d\mid L\left(0\right),A\left(0\right)=d_{A\left(0\right)}\left(V\right(0\left)\right)\right]\right)\hfill \\ \hfill D_2^{\ast }(d,P)& =\frac{I\left(\stackrel{‒}{A}\left(1\right)=d\left(A\right(0),V)\right)}{\prod _{j=0}^1{g}_{A\left(j\right)}\left(O\right)}\left(Y-E_P\left[Y\mid \stackrel{‒}{A}\left(1\right)=d\left(A\right(0),V),\stackrel{‒}{L}\left(1\right)\right]\right).\hfill \end{array}$$ (4)

Above (g_A(0), g_A(1)) is the intervention mechanism under the distribution P. We remind the reader that Y_d has the G-computation distribution from (1) so that:

$$E_P\left[Y_d\mid L\left(0\right),A\left(0\right)=d_{A\left(0\right)}\left(V\right(0\left)\right)\right]=E_P\left[E_P\left[Y\mid \stackrel{‒}{A}\left(1\right)=d\left(A\right(0),V),\stackrel{‒}{L}\left(1\right))\right]\mid L\left(0\right),A\left(0\right)=d_{A\left(0\right)}\left(V\right(0\left)\right)\right]$$

At times it will be convenient to write $D_k^{\ast }(d,Q^d,g)$ instead of $D_k^{\ast }(d,P)$, where Q^d represents both of the conditional expectations in the definitions of $D_1^{\ast }$ and the marginal distribution of L(0) under P and g represents the intervention mechanism under P. We will denote these conditional expectations under P₀ for a given rule d by $Q_0^d$. We will similarly at times denote D* (d, P) by D* (d, Q^d, g).

Whenever D* (P) does not contain an argument for a rule d, this D* (P) refers to the efficient influence curve of the parameter mapping Ψ for which Ψ(P) = E_PY_dP, where the optimal rule d_P under P is not treated as known. Not treating d_P as known means that d_P depends on the input distribution P in the mapping Ψ(P). The following theorem presents the efficient influence curve of Ψ at a distribution P. The main condition on this distribution P is that

$$\begin{array}{c}\hfill \underset{a_0\left(0\right)\in \{0,1\}}{\max }{\mathrm{Pr}}_P\left({\stackrel{‒}{Q}}_2\left(\left(a_0\right(0),1),V_{a\left(0\right)=\left(a_0\right(0),1)}\right)=0\right)=0\hfill \\ \hfill {\mathrm{Pr}}_P\left({\stackrel{‒}{Q}}_1\left(V\right(0\left)\right)=0\right)=0,\hfill \end{array}$$ (5)

where ${\stackrel{‒}{Q}}_2$ and ${\stackrel{‒}{Q}}_1$ are defined analogously to ${\stackrel{‒}{Q}}_{20}$ and ${\stackrel{‒}{Q}}_{10}$ in Theorem 1 with the expectations under P₀ replaced by expectations under P. That is, we assume that each of the blip functions under P is nowhere zero with probability 1. Distributions that do not satisfy this assumption have been referred to as “exceptional laws” [32, 52]. These laws are indeed exceptional when one expects that treatment will have a beneficial or harmful effect in all V-strata of individuals. When one only expects that treatment will have an effect on outcome in some but not all strata of individuals then this assumption may be violated. We will make this assumption about P₀ for all subsequent asymptotic linearity results about E_P0Y_d0, and we will assume a weaker but still not completely trivial assumption for the data adaptive target parameters in Sections 6 and 7.

Theorem 2. Suppose $\mathcal{P}\in \mathcal{M}$ such that Pr_P(|Y| <M) = 1 for some M <∞ and the positivity assumption (2) and (5). Then the parameter $\Psi :\mathcal{M}\to \text{IR}$ is pathwise differentiable at P with canonical gradient given by

$$D^{\ast }\left(P\right)\equiv D^{\ast }(d_P,P)=\sum _{k=0}^2{D}_k^{\ast }(d_P,P).$$

That is, D*(P) equals the efficient influence curve D*(d_P, P) for the parameter ψ_d(P)≡E_PY_d at the V-optimal rule d = d_P, where ψ_d treats d as given.

The above theorem is proved as Theorem 8 in van der Laan and Luedtke [48] so the proof is omitted here.

We will at times denote D*(P) by D*(Q, g), where Q represents Q^d_P, along with portions of the likelihood which suffice to compute the V-optimal rule d_P. We denote d_P by d_Q when convenient. We explore which parts of the likelihood suffice to compute the V-optimal rule in our companion paper, though Theorem 1 shows that ${\stackrel{‒}{Q}}_{20}$ and ${\stackrel{‒}{Q}}_{10}$ suffice for d₀ (and analogous functions suffice for a more general d_P). We have the following property of the efficient influence curve, which will provide a fundamental ingredient in the analysis of the TMLE presented in the next section.

Theorem 3. Let d_Q be the V-optimal rule corresponding with Q. For any Q, g, we have

$$P_0{D}^{\ast }(Q,g)=\Psi \left(Q_0\right)-\Psi \left(Q\right)+R_{1d_Q}\left(Q^{d_Q},Q_0^{d_Q},g,g_0\right)+R_2(Q,Q_0)$$

where for all $d\in \mathcal{D}$

$$R_{1d}(Q^d,Q_0^d,g,g_0)\equiv P_0{D}^{\ast }(d,Q^d,g)-\left({\Psi }_d\right(Q_0^d)-{\Psi }_d(Q^d\left)\right),$$

ψ_d(P) = E_PY_d is the statistical target parameter that treats d as known, and $D^{\ast }(d,Q_0^d,g_0)$ is the efficient influence curve of ψ_d at P₀ as given in Theorem 2. In addition,

$$\begin{array}{cc}\hfill R_2(Q,Q_0)& \equiv {\Psi }_{d_Q}\left(Q_0^{d_Q}\right)-{\Psi }_{d_0}\left(Q_0^{d_0}\right)\hfill \\ \hfill & =E_{P_0}\left(d_{Q,A\left(0\right)}-d_{0,A\left(0\right)}\right)\left(V\left(0\right)\left){\stackrel{‒}{Q}}_{10}\right(V\left(0\right)\right)+E_{P_0}\left(d_{Q,A\left(1\right)}-d_{0,A\left(1\right)}\right)\left((0,1),V_{(0,1)}\left(1\right)\right){\stackrel{‒}{Q}}_{20}\left((0,1),V_{(0,1)}\left(1\right)\right)\hfill \\ \hfill & \equiv R_{2A\left(0\right)}(Q,Q_0)+R_{2A\left(1\right)}(Q,Q_0).\hfill \end{array}$$

From the study of the statistical target parameter ψ_d in van der Laan and Gruber [41], we know that $P_0{D}^{\ast }(d,Q^d,g)={\Psi }_d\left(Q_0^d\right)-{\Psi }_d\left(Q^d\right)+R_{1d}(Q^d,Q_0^d,g,g_0)$, where R_1d is a closed form second-order term involving integrals of differences $Q^d-Q_0^d$ times differences g − g₀.

The following lemma bounds R₂. We note that this lemma, which concerns how well we can estimate d₀ rather than how well we can make inference about E_P0Y_d0, does not require condition (5) to hold. We showed in Theorem 1 that knowing the blip functions ${\stackrel{‒}{Q}}_{10}$ and ${\stackrel{‒}{Q}}_{20}$ suffices to define the optimal rule d₀. For general Q, we will let ${\stackrel{‒}{Q}}_1$ and ${\stackrel{‒}{Q}}_2$ represent the blip functions under this parameter mapping.

Lemma 1. Let R₂ be as in Theorem 3. Let P_0,(0,1) represent the static intervention-specific G-computation distribution where treatment (0, 1) is given at the first time point. Suppose there exist some β₁, β₂ > 1 such that:

$$\begin{array}{c}\hfill E_{P_0}\left[{\mid {\stackrel{‒}{Q}}_{10}\left(V\right(0\left)\right)\mid }^{-{\beta }_1}I\left(\mid {\stackrel{‒}{Q}}_{10}\left(V\right(0\left)\right)\mid >0\right)\right]<\infty \hfill \\ \hfill E_{P_{0,(0,1)}}\left[{\mid {\stackrel{‒}{Q}}_{20}\left((0,1),V_{(0,1)}\left(0\right)\right)\mid }^{-{\beta }_2}I\left(\mid {\stackrel{‒}{Q}}_{20}\left(\right(0,1),V_{(0,1)}(0\left)\right)\mid >0\right)\right]<\infty ,\hfill \end{array}$$ (6)

where the expression in each expectation is taken to be 0 when the indicator is 0. Fix p ∈ (1, ∞] and define h : (1, ∞] × (1, ∞) as the function for which $h(p,\beta )=\frac{p(\beta +1)}{p+\beta }$ when p<∞ and h(p, β) = β + 1 otherwise. Then:

$$\begin{array}{c}\hfill R_{2A\left(0\right)}(Q,Q_0)\le K_1{\Vert {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}\Vert }_{p,P_0}^{h(p,{\beta }_1)}\hfill \\ \hfill R_{2A\left(1\right)}(Q,Q_0)\le K_2{\Vert {\stackrel{‒}{Q}}_2-{\stackrel{‒}{Q}}_{20}\Vert }_{p,P_{0,(0,1)}}^{h(p,{\beta }_2)},\hfill \end{array}$$

where ∥·∥_p,P denotes the L_p,P norm for the distribution P and K₁, K₂ ≥ 0 are finite constants that respectively rely on p, P₀, β₁ and p, P_0,(0,1), β₂.

The conditions in (6) are moment bounds which ensure that ${\stackrel{‒}{Q}}_{10}$ and ${\stackrel{‒}{Q}}_{20}$ do not put too much mass around zero. To get the tightest bound, we should always choose β₁, β₂ to be as large as possible. We remind the reader that convergence in L_p,P implies convergence in L_q,P for all distributions P and 1 ≤ q ≤ p ≤ ∞. Hence there is a trade-off between the chosen bounding norm, L_p,P, and the rate we need to obtain with respect to that norm so that the term can be expected to be of order n^−1/2. See Table 1 for some examples of rates of convergence that suffice to give R_2A(0) = o_P0 (n^−1/2).

Table 1.

Convergence rates of estimators of ${\stackrel{‒}{Q}}_{10}$ which suffice for R_2A(0) to be o_P0(n^−1/2) according to Lemma 1. The higher the moments of ${\stackrel{‒}{Q}}_{10}^{-1}$ that are finite, the slower the estimator needs to converge. It is of course preferable to have an estimator which converges according to the P_o essential supremum than just in L_2,P0 but whether or not there is convergence in L_∞,P0 depends on the estimator used and the underlying distribution P₀.

p	β 1	Sufficient Lp,P0 convergence rate
2	1	oP0 (n−3/8)
	2	oP0 (n−1/3)
	β1 large	oP0 (n−(1/4+ε))for small ε>0
4	1	oP0 (n−5/16)
	2	oP0 (n−1/4)
	β1 large	oP0 (n−(1/8+ε))for small ε>0
∞	1	oP0 (n−1/4)
	2	oP0 (n−1/6)
	β1 large	$O_{P_0}\left(n^{-\frac{1}{2({\beta }_1+1)}}\right)$

Using the upper bound on ${\stackrel{‒}{Q}}_{10}$ and applying Cauchy-Schwarz inequality to eq. (15) in the proof of the lemma shows that:

$$R_{2A\left(0\right)}(Q,Q_0)\le {\Vert {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}\Vert }_{2,P_0}\sqrt{P{r}_{P_0}\left(0<\mid {\stackrel{‒}{Q}}_{10}\mid <\mid {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}\mid \right).}$$

Hence R_2A(0) = o_P0 (n^−1/2) without any moment condition when $\Vert {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}{\Vert }_{2,P_0}=O_{P_0}\left(n^{-1∕2}\right)$, which occurs when one has correctly specified a parametric model for ${\stackrel{‒}{Q}}_{10}$. In general it is unlikely that one can correctly specify a parametric model for ${\stackrel{‒}{Q}}_{10}$. In these cases, Lemma 1 shows that the term R_2A(0) will still be o_P0 (n^−1/2) if a moment condition holds and ${\stackrel{‒}{Q}}_{10}$ is estimated at a sufficient rate. The analogue holds for ${\stackrel{‒}{Q}}_{20}$.

The bounds given in Lemma 1 are loose. It is not in general necessary to estimate the blip functions ${\stackrel{‒}{Q}}_{10}$ and $Q_{20}$ correctly, only their signs. As an extreme example of the looseness of the bounds, one can have that ${\text{inf}}_{v\left(0\right)}\mid {\stackrel{‒}{Q}}_{1n}\left(v\right(0\left)\right)-{\stackrel{‒}{Q}}_{10}\left(v\right(0\left)\right)\mid \to \infty$ as n → ∞ and still have that R_2A(0)(Q, Q_n) = 0 for all n. Nonetheless, these bounds give interpretable sufficient conditions under which the term R₂ converges faster than a root-n rate. We consider methods that do not directly estimate the blip functions in our companion paper.

4 TMLE of the mean outcome under V-optimal rule

Throughout this and the next section we assume that condition (5) holds at P₀. Our proposed TMLE is to first estimate the optimal rule d₀, giving us an estimated rule d_n(A(0),V) = d_n,A(0)(V(0)),d_n,A(1)(A(0),V(1)), and subsequently apply the TMLE of EY_d for a fixed rule d at d_n = d_n as presented in van der Laan and Gruber [41]. This TMLE is an analogue of the double robust estimating equation method presented in Bang and Robins [36]: see also Petersen et al. [40] for a generalization of the TMLE to marginal structural models for dynamic treatments. In a companion paper we describe a data adaptive estimator of d₀. In this paper we take d_n as given. We review the TMLE for ψ_d(P₀) = E_P0Y_d at a fixed rule d in “TMLE of the mean outcome under a given rule” in Appendix B. Observations which are only partially observed due to right censoring do not cause a problem for the TMLE. In particular, the TMLE only uses individuals who are not right censored at the first or second time point to obtain initial estimates of E_P0[Y_d|A(0) = d_A(0)(V(0)),L(0)] and $E_{P_0}[Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$ in (4), respectively. See the appendix for details.

Here we note some of the key properties of the TMLE. Let $Q_n^{d_n\ast }$ consist of the empirical distribution Q_L(0),n of L(0), a regression function $l\left(0\right)\mapsto E_n^{\ast }[Y_d\mid L(0)=l(0\left)\right]$ that estimates E_P0[Y_d|L(0)], and a regression function

$$\left(a\left(0\right),\stackrel{‒}{l}\left(1\right)\right)\mapsto E_n^{\ast }\left[Y\mid \stackrel{‒}{A}\left(1\right)=d\left(a\right(0),v),\stackrel{‒}{L}\left(1\right)=\stackrel{‒}{l}\left(1\right)\right]$$

that estimates $E_{P_0}[Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$, where we note that v is a function of $\stackrel{‒}{l}\left(1\right)$. In the appendix we describe our proposed algorithm to get the estimates in $Q_n^{d_n\ast }$. The proposed TMLE for ψ₀ = E_P0Y_d0 is given by

$${\psi }_{d_n,n}^{\ast }={\Psi }_{d_n}\left(Q_n^{d_{n^{\ast }}}\right)=\frac{1}{n}\sum _{i=1}^n{E}_n^{\ast }\left[Y_{d_n}\mid L\left(0\right)=l{\left(0\right)}_i\right],$$

where we have applied the TMLE in the appendix to the case where d = d_n, treating d_n as known. Note that ${\Psi }_{d_n}\left(Q_n^{d_n\ast }\right)$ is a plug-in estimator in that it is obtained by plugging $Q^{d_n^{\ast }}$ into the parameter mapping Q^d ↦ ψ_d(Q^d) for d = d_n. We expect our plug-in estimator to give reasonable estimates in finite samples because it naturally respects the constraints of our model. In the next section we show that this estimator also enjoys many desirable asymptotic properties.

Recall that D^*(d, Q^d, g) is the efficient influence curve for the target parameter E_P0Y_d which treats d as fixed, and Theorem 2 showed that $D^{\ast }(d_0,Q_0^{d_0},g_0)$ is the efficient influence curve of the target parameter EY_d0 where d₀ is the V-optimal rule. The TMLE $(d_n,Q_n^{d_n\ast })$ described in the appendix solves the efficient influence curve estimating equation:

$$P_n{D}^{\ast }\left(d_n,Q_n^{d_{n^{\ast }}},g_n\right)=0.$$ (7)

Further, one can show using standard M-estimator analysis that the targeted $Q_n^{d_n\ast }$ proposed in the appendix maintains the same rate of convergence as the initial estimator $Q_n^{d_n}$ under very mild conditions. We do not concern ourselves with these conditions in this paper, and will instead state all conditions directly in terms of $Q_n^{d_n\ast }$. The above will be a key ingredient in proving the asymptotic linearity of the TMLE for ψ₀ = E_P0Y_d0.

5 Asymptotic efficiency of the TMLE of the mean outcome under the V-optimal rule

We now wish to analyze the TMLE ${\psi }_n^{\ast }={\Psi }_{d_n}\left(Q_n^{d_n\ast }\right)$ of ${\psi }_0={\Psi }_{d_0}\left(Q_0^{d_0}\right)=\Psi \left(Q_0\right)$. We first give a representation that will allow us to prove the asymptotic linearity of the TMLE under conditions. The result allows $Q_n^{d_n\ast }$ to be misspecified, even though the intervention mechanism g₀ and the rule d_n are assumed to be consistent for g₀ and d₀, respectively.

Theorem 4. Assume Y ∈ [0,1], the strong positivity assumption, condition (5) at p₀,$D_n^{\ast }\equiv D^{\ast }(d_n,Q_n^{d_n\ast },g_n)$ falls in a P₀-Donsker class with probability tending to 1, $P_0{\{D_n^{\ast }-D^{\ast }(d_0,Q^{d_0},g_0\left)\right\}}^2$ converges to zero in probability for some Q^d0, and

$$R_2(Q_n,Q_0)=O_{P_0}(1∕\sqrt{n}),$$

where R₂ is defined in Theorem 3 and an upper bound is established in Lemma 1. Then

$${\Psi }_n^{\ast }-{\Psi }_0=(P_n-P_0)D^{\ast }(d_0,Q^{d_0},g_0)+R_{1d_n}(Q_n^{d_n},Q_0^{d_n},g_n,g_0)+O_{P_0}\left(n^{-\frac{1}{2}}\right),$$ (8)

where R_1d is defined in Theorem 3.

The proof of the above theorem, which is given in the appendix, makes use of the fact that the TMLE satisfies (7). We now give two sets of conditions which control the remainder term R_1dn in (8) to prove the asymptotic linearity of the TMLE. The first result is an immediate consequence of the fact that $R_{1d_n}(Q_n^{d_n},Q_0^{d_n},g_n,g_0)=0$ whenever g_n = g₀.

Corollary 1. Suppose the conditions of Theorem 4 further suppose that g_n = g₀ (i.e., RCT). Then:

$${\Psi }_n^{\ast }-{\Psi }_0=(P_n-P_0)D^{\ast }(d_0,Q^{d_0},g_0)+O_{P_0}\left(n^{-1∕2}\right)$$

That is, ${\psi }_n^{\ast }$ is asymptotically linear with influence curve D^* (d₀, Q^d0, g₀).

The next corollary is more general in that it applies to situations where the intervention mechanism g₀ is estimated from the data. The above result emerges as a special case.

Corollary 2. Suppose all of the conditions of Theorem 4 hold, and that

$$R_{1d_n}\left(Q_n^{d_{n^{\ast }}},Q_0^{d_n},g_n,g_0\right)-R_{1d_n}\left(Q^{d_n},Q_0^{d_n},g_n,g_0\right)=O_{P_0}(1∕\sqrt{n})$$

for some Q^d_n. In addition, we assume the following asymptotic linearity condition on a smooth functional of g_n:

$$R_{1d_n}\left(Q^{d_n},Q_0^{d_n},g_n,g_0\right)=(P_n-P_0)D_g\left(P_0\right)+O_{P_0}(1∕\sqrt{n}),$$ (9)

for some function $D_g\left(P_0\right)\left(0\right)\in L_0^2\left(P_0\right)\equiv \{h:P_{0}h=0,P_0{h}^2<\infty \}$. Then,

$${\Psi }_n^{\ast }-{\Psi }_0=(P_n-P_0)\left\{D^{\ast }\right(d_0,Q^{d_0},g_0)+D_g(P_0\left)\right\}+O_{P_0}(1∕\sqrt{n}).$$ (10)

If it is also know that g_n is an MLE of g₀ according to a correctly specified model G for g₀ with tangent space T_g(P₀) at P₀, then (9) holds with

$$D_g\left(P_0\right)=-\Pi \left(D^{\ast }\right(d_0,Q^{d_0},g_0)\mid T_g(P_0\left)\right),$$ (11)

where Π(·T_g(P₀)) denotes the projection operator onto $T_g\left(P_0\right)\subset L_0^2\left(P_0\right)$ in the Hilbert space $L_0^2\left(P_0\right)$.

Equation (11) is a corollary of Theorem 2.3 of van der Laan and Robins [34]. The rest of the theorem is the result of a simple rearrangement of terms, so the proof is omitted.

Condition (9) is trivially satisfied in a randomized clinical trial without missingness, where we can take g_n = g₀ and thus D_g(P₀) is the constant function 0. Nonetheless, (11) suggests that it would be better to estimate g₀ using a parametric model that contains the true (known) intervention mechanism. For example, at each time point one may use a main terms linear logistic regression with treatment and covariate histories as predictors. If $Q_n^{d_n}$ consistently estimates $Q_0^{d_0}$, then D* (d₀, Q^d₀, g₀) is orthogonal to T_g(P₀) and hence the projection in (11) is the constant function 0. Otherwise the projection will decrease the variance of ${\psi }_n^{\ast }-{\psi }_0$ without affecting asymptotic bias, thereby increasing the asymptotic efficiency of the estimator. One can then use an empirical estimate of the variance of D*(d₀, Q^d₀, g₀) to get asymptotically conservative confidence intervals for ψ₀.

5.1 Asymptotic linearity of TMLE in a SMART setting

Suppose the data is generated by a sequential RCT and there is no missingness so that g₀ is known. Further suppose that (5) holds at P₀, that is, that treating at each time point has either a positive or negative effect with probability 1, regardless of the choice of the regimen at earlier time points. In addition, assume that V(0) and V(1) are both univariate scores, and assume condition (3) so that the optimal rule d_0,A(1) based on (A(0), V(0), V(1)) is the same as the optimal rule d_0,A(1) based on A(0), V(1): for example, V(1) is the same score as V(0) but measured at the next time point, so that it is reasonable to assume that an effect of V(0) on Y will be fully blocked by V(1). Suppose we want to use the data of the RCT to learn the V-optimal rule d₀ and provide statistical inference for E_P0Y_d0. Further suppose that the moment conditions in Lemma 1 hold with β₁ = β₂ = 2. Since both V(0) and V(1) are one-dimensional, using kernel smoothers or sieve-based estimation to generate a library of candidate estimators for the sequential loss-based super-learner of the blip functions (${\stackrel{‒}{Q}}_{10},{\stackrel{‒}{Q}}_{20}$) described in our companion paper, we can obtain an estimator ${\stackrel{‒}{Q}}_n=({\stackrel{‒}{Q}}_{1n},{\stackrel{‒}{Q}}_{2n})$ of ${\stackrel{‒}{Q}}_0=({\stackrel{‒}{Q}}_{10},{\stackrel{‒}{Q}}_{20})$ that converges in L₂ at a rate such as n^−2/5 under the assumption that ${\stackrel{‒}{Q}}_{10},{\stackrel{‒}{Q}}_{20}$ are continuously differentiable with a uniformly bounded derivative, or at a better rate under additional smoothness assumptions. As a consequence, in this case R₂(Q_n, Q₀) = O_P0 (n^−3/5) = O_Po(n^−1/2) by Lemma 1. As a consequence, all conditions of Theorem 4 hold, and it follows that the proposed TMLE is asymptotically linear with influence curve D* (d₀, Q^d₀, g₀), where Q^d₀ is the possibly misspecified limit of Q^d_n* in the TMLE. To conclude, sequential RCTs allow us to learn V-optimal rules at adaptive optimal rates of convergence, and allow valid asymptotic statistical inference for E_P0 Y_d0. If V(j) is higher dimensional, then one will have to rely on enough smoothness assumptions on the blip functions and/or moment conditions on $1∕\mid {\stackrel{‒}{Q}}_{10}\mid$ and $1∕\mid {\stackrel{‒}{Q}}_{20}\mid$ from Lemma 1 in order to guarantee that $R_2(Q_n,Q_0)=o_{P_0}(1∕\sqrt{n})$.

If there is right censoring, then g₀ = g₀₁g₀₂ factors in a treatment mechanism g₀₁ and censoring mechanism g₀₂, where g₀₁ is known, but g₀₂ is typically not known. Having a lot of knowledge about how censoring depends on the observed past might make it possible to obtain a good estimator of g₀₂. In that case, the above conclusions still apply, but one now estimates the nuisance parameters of the loss function (e.g., one uses a double robust loss function in which g02 is replaced by an estimator, see our companion paper).

5.2 Statistical inference

Suppose one wishes to estimate the mean outcome under the optimal rule E_P0 Y_d0 and that (5) holds. Above we developed the TMLE ${\psi }_n^{\ast }$ for E_P0 Y_d0. By Corollary 1, if g_n = g₀ is known, this TMLE of ψ₀ is asymptotically linear with influence curve IC(P₀) = D* (d₀, Q^d₀, g₀). If g_n is an MLE according to a model with tangent space T_g(P₀), then the TMLE is asymptotically linear with influence curve

$$\mathit{IC}\left(P_0\right)-\Pi \left(\mathit{IC}\right(P_0)\mid T_g(P_0\left)\right),$$

so that one could use IC(P₀) as a conservative influence curve. Let IC_n be an estimator of this influence curve IC(P₀) obtained by plugging in the available estimates of its unknown components. The asymptotic variance of the TMLE ${\psi }_n^{\ast }$ of ψ₀ can now be (conservatively) estimated with

$${\sigma }_n^2=\frac{1}{n}\sum _{i=1}^n{\mathit{IC}}_n^2\left(O_i\right).$$

An asymptotic 95% confidence interval for ψ₀ is given by ${\psi }_n^{\ast }\pm 1.96{\sigma }_n∕\sqrt{n}$.

6 Statistical inference for mean outcome under data adaptively determined dynamic treatment

Let $\widehat{d}:\mathcal{M}\to \mathcal{D}$ be an estimator that maps an empirical distribution into an individualized treatment rule. See our companion paper for examples of possible estimators $\widehat{d}$. Let $d_n=\widehat{d}\left(P_n\right)$ be the estimated rule. Up until now we have been concerned with statistical inference for E_P0 Y_d0, where d₀ is the unknown V-optimal rule while d_n is a best estimator of this rule. As a consequence, statistical inference for E_P0 Y_d0 based on the TMLE relied on consistency of d_n to d₀, but also relied on the rate of convergence at which d_n converges to d₀, that is, $R_2(Q_n,Q_0)=o_{P_0}(1∕\sqrt{n})$. In this section we present statistical inference for the data adaptive target parameter

$${\Psi }_{0n}={\Psi }_{d_n}\left(P_0\right)=E_{P_0}{Y}_d{\mid }_{d=d_n}.$$

That is, we construct an estimator ${\widehat{\Psi }}_{\widehat{d}\left(P_n\right)}\left(P_n\right)$ of ${\Psi }_{\widehat{d}\left(P_n\right)}\left(P_0\right)$ and a confidence interval so that

$$\underset{n\to \infty }{\lim }{\text{Pr}}_{P_0}\left({\Psi }_{\widehat{d}\left(P_n\right)}\left(P_0\right)\in {\widehat{\Psi }}_{\widehat{d}\left(P_n\right)}\left(P_n\right)\pm 1.96\widehat{\sigma }\left(P_n\right)∕\sqrt{n}\right)=0.95,$$

where $\widehat{\sigma }\left(P_n\right)$ is a consistent estimator of the standard error of ${\widehat{\Psi }}_{\widehat{d}\left(P_n\right)}\left(P_n\right)$. Note that in this definition of the confidence interval the target parameter is itself also a random variable through the data P_n.

We do not assume that (5) holds in this section, but we do implicitly make the weaker assumption that d_n → d₁ for some $d_1\in \mathcal{D}$ in assumption (12) of Theorem 5. Statistical inference will be based on the same TMLE of ψ_d(P₀) at d = d_n, and our variance estimator will also be the same, but since the target is not ψ_d0 (P₀) but ψ_dn(P₀), there will be no need for d_n to even be consistent for d₀, let alone converge at a particular rate. As a consequence, this approach is particularly appropriate in cases where V is high dimensional so that it is not reasonable to expect that d_n converges to d₀ at the required rate. Another motivation for this data adaptive target parameter is that, even when statistical inference for E_P0 Y_d0 is feasible, one might be interested in statistical inference for the mean outcome under the concretely available rule d_n instead of under the unknown rule d₀.

As shown in the proof of Theorem 3, $P_0{D}^{\ast }(d_n,Q_n^{\ast },g_n)={\psi }_{0n}-{\psi }_n^{\ast }+R_{1d_n}(Q_n^{d_n\ast },Q_0^{d_n},g_n,g_0)$. Further, $P_n{D}^{\ast }(d_n,Q_n^{d_n\ast },g_n)=0$, which yields

$${\Psi }_n^{\ast }-{\Psi }_{0n}=(P_n-P_0)D^{\ast }(d_n,Q_n^{d_{n^{\ast }}},g_n)+R_{1d_n}(Q_n^{d_{n^{\ast }}},Q_0^{d_n},g_n,g_0).$$

This relation is key to the proof of the following theorem, which is analogous to Theorem 4. Note crucially that the theorem does not have any conditions on the remainder term R₂, nor does it require that d_n converge to the optimal rule d₀.

Theorem 5. Assume Y ∈ [0, 1]. Let $\widehat{d}\left(P_n\right)\in \mathcal{D}$ with probability tending to 1, and assume the strong positivity assumption. Let ψ_0n = ψ_dn(P₀) = E_P0 Y_d|d=dn be the data adaptive target parameter of interest. Let R_1d be as defined in Theorem 3.

Assume $D_n^{\ast }\equiv D^{\ast }(d_n,Q_n^{\ast },g_n)$ falls in a P₀ -Donsker class with probability tending to 1,

$$P_0{\{D_n^{\ast }-D^{\ast }(d_1,Q^{d_1},g_0\left)\right\}}^2=O_{P_0}\left(1\right)$$ (12)

for some $d_1\in \mathcal{D}$ and Q^d1. Then,

$${\Psi }_n^{\ast }-{\Psi }_{0n}=(P_n-P_0)D^{\ast }(d_1,Q^{d_1},g_0)+R_{1d_n}(Q_n^{d_{n^{\ast }}},Q_0^{d_n},g_n,g_0)+O_{P_0}\left(n^{-1∕2}\right).$$

If g_n = g₀ (i.e., RCT), then $R_{1d_n}(Q_n^{d_n\ast },Q_0^{d_n},g_n,g_0)=0$, so that ${\psi }_n^{\ast }$ is asymptotically linear with influence curve D* (d₁, Q, g₀).

The proof of the above theorem is nearly identical to the proof of Theorem 4 so is omitted. For general g_n, $R_{1d_n}(Q_n^{d_n\ast },Q_0^{d_n},g_n,g_0)=o_{P_0}\left(n^{-1∕2}\right)$ under an analogous second-order term condition to the one assumed in Corollary 1. As in Corollary 2, the asymptotic efficiency may improve (and will not worsen) when a known intervention mechanism is fit using a correctly specified parametric model. See Theorem 11 in our online technical report for details [47].

7 Statistical inference for the average of sample-split specific mean counterfactual outcomes under data adaptively determined dynamic treatments

Again let $\widehat{d}:\mathcal{M}\to \mathcal{D}$ be an estimator that maps an empirical distribution into an individualized treatment rule. Let B_n ∈ {0, 1}ⁿ denote a random vector for a cross-validation split, and for a split B_n, let $P_{n,B_n}^0$ be the empirical distribution of the training sample {i : B_n(i) = 0} and $P_{n,B_n}^1$ is the empirical distribution of the validation sample {i : B_n(i) = 1}. Consider a J-fold cross-validation scheme. In J-fold cross-validation, the data is split into J mutually exclusive and exhaustive sets of size approximately n/J uniformly at random. Each set is then used as the validation set once, with the union of all other sets serving as the training set. With probability 1/J, B_n has value 1 in all indices in validation set j ∈ {1; …, J} and 0 for all indices not corresponding to training set j.

In this section, we present a method that provides an estimator and statistical inference for the data adaptive target parameter

$${\tilde{\Psi }}_{0n}=E_{B_n}{\Psi }_{\tilde{d}\left(P_{n,B_n}^0\right)}\left(P_0\right).$$

Note that ${\tilde{\psi }}_{0n}$ is different from the data adaptive target parameter ψ_0n presented in the previous section. In particular, this target parameter is defined as the average of data adaptive parameters, where the data adaptive parameters are learned from the training samples of size approximately n/J. In the previous section, the data adaptive target parameter was defined as the mean outcome under the rule d_n which was estimated on the entire data set. Again the target parameter is a random quantity that relies on the sample of size n.

One applies the estimator $\widehat{d}$ to each of the J training samples, giving a target parameter value ${\Psi }_{\widehat{d}\left(P_{n,B_n}^0\right)}\left(P_0\right)$, and our target parameter ${\tilde{\psi }}_{0n}$ is defined as the average across these J target parameters. Below we present a CV-TMLE ${\tilde{\psi }}_n^{\ast }$ of this data adaptive target parameter ${\tilde{\psi }}_{0n}$. As in the previous section, we will be able to establish statistical inference for our estimate ${\tilde{\psi }}_n^{\ast }$ without requiring that the estimated rules converge to d₀, nor any rate condition on the estimated rules. Unlike the asymptotic linearity results in all previous sections, the results in this section do not rely on an empirical process condition (i.e., Donsker class condition). That means we obtain valid asymptotic statistical inference under essentially no conditions in a sequential RCT, even when d_n is a highly data adaptive estimator of a V-optimal rule for a possibly high dimensional V. Under a consistency and rate condition (but no empirical process condition) on d_n, we also get inference for E_P0Y_d0.

The next subsection defines the general CV-TMLE for data adaptive target parameters. We subsequently present an asymptotic linearity theorem allowing us to construct asymptotic 95% confidence intervals.

7.1 General description of CV-TMLE

Here we give a general overview of the CV-TMLE procedure. In “CV-TMLE of the mean outcome under data adaptive V-optimal rule” in Appendix B we present a particular CV-TMLE which satisfies all of the properties described in this section. Denote the realizations of B_n with j = 1, …, J, and let $d_{nj}=\widehat{d}\left(P_{n,j}^0\right)$ for some estimator of the optimal rule $\widehat{d}$. Let

$$\left(a\right(0),\stackrel{‒}{l}(1\left)\right)\mapsto E_{nj}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(a\left(0\right),v),\stackrel{‒}{L}(1)=\stackrel{‒}{l}(1\left)\right]$$

represent an initial estimate of $E_{P_0}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$ based on the training sample j. Similarly, let l(0) ↦ E_nj[Y_dnj |L(0) = l(0)] represent an initial estimate of E_P0[Y_dnj |L(0)] based on the training sample j. Finally, let Q_L(0),nj represent the empirical distribution of L(0) in validation sample j. We then fluctuate these three regression functions using the following submodels:

$$\begin{array}{cc}\hfill & \left\{E_{nj}^{\left({\epsilon }_2\right)}\right[Y\mid \stackrel{‒}{A}\left(1\right)=d_{nj}\left(a\right(0),v),\stackrel{‒}{L}\left(1\right)=\stackrel{‒}{l}\left(1\right)]:{\epsilon }_2\in \mathbb{R}\}\hfill \\ \hfill & \left\{E_{nj}^{\left({\epsilon }_2\right)}\right[Y_{d_{nj}}\mid L\left(0\right)=l\left(0\right)]:{\epsilon }_1\in \mathbb{R}\}\hfill \\ \hfill & \{Q_{L\left(0\right),nj}^{\left({\epsilon }_0\right)}:{\epsilon }_0\in \mathbb{R}\}\hfill \end{array}$$

where these submodels rely on an estimate g_nj of g₀ based on training sample j and are such that:

$$\begin{array}{cc}\hfill E_{nj}^{\left(0\right)}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(a\left(0\right),v),\stackrel{‒}{L}(1\left)\right]& =E_{nj}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(a\left(0\right),v),\stackrel{‒}{L}(1\left)\right]\hfill \\ \hfill E_{nj}^{\left(0\right)}[Y_{d_{nj}}\mid L(0\left)\right]& =E_{nj}[Y_{d_{nj}}\mid L(0\left)\right]\hfill \\ \hfill Q_{L\left(0\right),nj}^{\left(0\right)}& =Q_{L\left(0\right),nj}.\hfill \end{array}$$

Let $Q_{nj}^{d_{nj}}\left(\epsilon \right)$ represent the parameter mapping that gives the three regression functions above fluctuated by ε≡(ε₀, ε₁, ε₂). For a fixed ε, $Q_{nj}^{d_{nj}}\left(\epsilon \right)$ only relies on $P_{nj}^1$ through the empirical distribution of L(0) in validation sample j. Let ϕ be a valid loss function for $Q_0^d$ so that $Q_0^d={\text{arg min}}_{Q^d}{P}_0\varphi \left(Q^d\right)$, and let ϕ and the submodels above satisfy

$$D^{\ast }(d,Q^d,g)\in \langle \frac{d}{d\epsilon }\varphi \left(Q^d\right(\epsilon \left)\right){\mid }_{\epsilon =0}\rangle ,$$

where 〈f〉 = {Σ_j β_jf_j : β} denotes the linear space spanned by the components of f. We choose ε_n to minimize $P_n^1\varphi \left(Q_{nj}^{d_{nj}}\right(\epsilon \left)\right)$ over $\epsilon \in {\mathbb{R}}^3$. We then define the targeted estimate $Q_{nj}^{d_{nj}\ast }\equiv Q_{nj}^{d_{nj}}\left({\epsilon }_n\right)$ of $Q_0^{d_{nj}}$. We note that $Q_{nj}^{d_{nj}\ast }$ maintains the rate of convergence of Q_nj under mild conditions that are standard to M-estimator analysis. The key property that we need from the ε_n and the corresponding update $Q_{nj}^{d_{nj}\ast }$ is that it (approximately) solves the cross-validated empirical mean of the efficient influence curve:

$$E_{B_n}{P}_{n,B_n}^1{D}^{\ast }\left(d_{nj},Q_{nj}^{d_{n{j}^{\ast }}},g_{nj}\right)=O_{P_0}(1∕\sqrt{n}).$$ (13)

The CV-TMLE implementation presented in the appendix satisfies this equation with $o_{P_0}(1∕\sqrt{n})$ replaced by the 0. The proposed estimator of ${\tilde{\psi }}_{0n}$ is given by

$${\tilde{\psi }}_n^{\ast }\equiv E_{B_n}{\Psi }_{d_{nj}}\left(Q_{nj}^{{d_{nj}}^{\ast }}\right).$$

In the current literature we have referred to this estimator as the CV-TMLE [53–56]. We give a concrete CV-TMLE algorithm for ${\tilde{\psi }}_n^{\ast }$ in “CV-TMLE of the mean outcome under data adaptive V-optimal rule” in Appendix B, but note that other CV-TMLE algorithms can be derived using the approach in this section for different choices of loss function ϕ and submodels.

7.2 Statistical inference based on the CV-TMLE

We now proceed with the analysis of this CV-TMLE ${\tilde{\psi }}_n^{\ast }$ of ${\tilde{\psi }}_{0n}$. We first give a representation theorem for the CV-TMLE that is analogous to Theorem 5.

Theorem 6. Let g_nj and d_nj represent estimates of g₀ and d₀ based on training sample j. Let $Q_{nj}^{d_{nj}\ast }$ represent a targeted estimate of $Q_0^{d_{nj}}$ as presented in Section 7.1 so that $Q_{nj}^{d_{nj}\ast }$ satisfies (13). Let R_1d be as in Theorem 3. Further suppose that the supremum norm of max_j $D^{\ast }(d_{nj},Q_{nj}^{d_{nj}\ast },g_{nj})$ is bounded by some M <∞ with probability tending to 1, and that

$$\underset{j\in \{1,\dots ,J\}}{\max }{P}_0{\left\{D^{\ast }\left(d_{nj},Q_{nj}^{d_{n{j}^{\ast }}},g_{nj}\right)-D^{\ast }(d_1,Q^{d_1},g)\right\}}^2\to 0inprobability$$

for some $d_1\in \mathcal{D}$ and possibly misspecified Q^d₁ and g. Then:

$${\tilde{\psi }}_n^{\ast }-{\tilde{\psi }}_{0n}=(P_n-P_0)D^{\ast }(d_1,Q^{d_1},g^{d_1})+\frac{1}{J}\sum _{j=1}^J{R}_{1d_{nj}}\left(Q_{nj}^{d_{n{j}^{\ast }}},Q_0^{d_{nj}},g_{nj},g_0\right)+O_{P_0}\left(n^{-1∕2}\right).$$

Note that d₁ in the above theorem need not be the same as the optimal rule d₀, though later we will discuss the desirable special case where d₁ = d₀. The above theorem also does not require that g₀ is known, or even that the limit of our intervention mechanisms g is equal to g₀. Nonetheless, we get the following asymptotic linearity result when g = g₀ and g_nj satisfies an asymptotic linearity condition on a smooth functional of g_nj.

Corollary 3. Suppose the conditions from Theorem 6 hold with g = g₀. Further suppose that:

$$\frac{1}{J}\sum _{j=1}^J\left(R_{1d_{nj}}\left(Q_{nj}^{d_{n{j}^{\ast }}},Q_0^{d_{nj}},g_{nj},g_0\right)-R_{1d_{nj}}\left(Q^{d_{nj}},Q_0^{d_{nj}},g_{nj},g_0\right)\right)=o_{P_0}\left(n^{-1∕2}\right),$$

for some Q^d_nj and that:

$$\frac{1}{J}\sum _{j=1}^J{R}_{1d_{nj}}\left(Q^{d_{n{j}^{\ast }}},Q_0^{d_{nj}},g_{nj},g_0\right)=(P_n-P_0)D_g\left(P_0\right)+O_{P_0}\left(n^{-1∕2}\right).$$ (14)

We can conclude that:

$${\tilde{\psi }}_n^{\ast }-{\tilde{\psi }}_{0n}=(P_n-P_0)\left(D^{\ast }\right(d_1,Q^{d_1},g_0)+D_g(P_0\left)\right)+O_{P_0}\left(n^{-1∕2}\right).$$

The proof of the above result is just a rearrangement of terms so is omitted. Consider our setting. Suppose g₀ is known so we can have that g_nj = g₀ for all j. Consider the estimator

$${\sigma }_n^2=\frac{1}{J}\sum _{j=1}^J{P}_{n,j}^1{\left\{D^{\ast }\left(d_{nj},Q_{nj}^{d_{n{j}^{\ast }}},g_{nj}\right)\right\}}^2$$

of the asymptotic variance ${\sigma }_0^2=P_0{\left\{D^{\ast }\right(d_1,Q^{d_1},g_0\left)\right\}}^2$ of the CV-TMLE ${\tilde{\psi }}_n^{\ast }$. An asymptotic 95% confidence interval for ${\tilde{\psi }}_{0n}$ is given by ${\tilde{\psi }}_n^{\ast }\pm 1.95{\sigma }_n∕\sqrt{n}$. This same variance estimator and confidence interval can be used for the case that g₀ is not known and each g_nj is an MLE of g₀ according to some model. In that case, it is an asymptotically conservative confidence interval (analogous to eq. (11) applied to Corollary 3).

Now consider the case where d₁ from the above theorem is equal to the optimal rule d₀ and condition (5) holds. For simplicity, also assume that g₀ is known and g_nj = g₀. Then R_1dnj is equal to 0 for all j, so Theorem 6 shows that the CV-TMLE for ${\tilde{\psi }}_{0n}$ is asymptotically linear with influence curve $D^{\ast }(d_1,Q^{d1},g_0)=D^{\ast }(d_0,Q^{d_0},g_0)$. If

$${\tilde{\psi }}_{0n}-{\psi }_0=\frac{1}{J}\sum _{j=1}^J{R}_2(Q_{nj},Q_0)$$

is second order, that is, o_P0 (n^−1/2), where Q_nj is analogous to Q_n but only estimated on the training sample j, then the CV-TMLE is consistent and asymptotically normal estimator of the mean outcome under the optimal rule. If $Q^{d_0}=Q_0^{d_0}$, then the CV-TMLE is also asymptotically efficient among all regular asymptotically linear estimators. One can apply bounds like those in Lemma 1 for each of the J terms above to understand the behavior of ${\tilde{\psi }}_{0n}-{\psi }_0$. Note crucially that this result does not rely on the restrictive empirical process conditions used in the previous sections, although it relies on a consistency and rate condition for asymptotic linearity with respect to the non-data adaptive parameter E_P0Y_d0.

8 Simulation methods

We start by presenting two single time point simulations. In earlier technical reports we directly describe the single time point problem [47, 48]. Here, we instead note that a single time point optimal treatment is a special case of a two time point treatment when only the second treatment is of interest. In particular, we can see this by taking L(0) = V(0) = ∅, estimating ${\stackrel{‒}{Q}}_{2,0}$ without any dependence on a(0), and correctly estimating ${\stackrel{‒}{Q}}_{1,0}$ with the constant function zero. We note that, in this one time point formulation, we do not need (5) to hold for ${\stackrel{‒}{Q}}_{10}$, so it may be more natural to view the single time point problem directly and use the single time point pathwise differentiability result in Theorem 2 of van der Laan and Luedtke [48]. We can then let I(A(0) = d_n,A(0)(V(0))) = 1 for all A(0), V(0) wherever the indicator appears in our calculations. Because the first time point is not of interest, we only describe the second time point treatment mechanism for this simulation. We refer the interested reader to the earlier technical report for a thorough discussion of the single time point case. We then present a two time point data generating distribution to show the effectiveness of our proposed method in the longitudinal setting.

8.1 Data

8.1.1 Single time point

We simulate 1,000 data sets of 1,000 observations from an RCT without missingness. We have that:

$$\begin{array}{c}\hfill L_1\left(1\right),L_2\left(1\right),L_3\left(1\right),L_4\left(1\right)\mid A\left(0\right)\stackrel{iid}{~}N(0,1)\hfill \\ \hfill A_1\left(1\right)\mid A\left(0\right)~\text{Bern}(1∕2)\hfill \\ \hfill A_2\left(1\right)\mid A_1\left(1\right),A\left(0\right)~\text{Bern}\left(1\right)\hfill \end{array}$$

$$\mathrm{logit}{E}_{P_0}[Y\mid \stackrel{‒}{A}(1),\stackrel{‒}{L}(1),H=0]=1-L_1{\left(1\right)}^2+3L_2\left(1\right)+A_1\left(1\right)\left(5L_3{\left(1\right)}^2-4.45\right)$$

$$\mathrm{logit}{E}_{P_0}[Y\mid \stackrel{‒}{A}(1),\stackrel{‒}{L}(1),H=1]=-0.5-L_3\left(1\right)+2L_1\left(1\right)L_2\left(1\right)+A_1\left(1\right)(3\mid L_2(1)\mid -1.5)$$

where Y is a Bernoulli random variable and H is an unobserved Bern(1/2) variable independent of $\stackrel{‒}{A}\left(1\right)$, $\stackrel{‒}{L}\left(1\right)$. The above distribution was selected so that the mean outcomes under static treatments (treating everyone or no one at the second time point) have approximately the same mean outcome of 0.464.

We consider two choices for V(1). For the first we consider V(1) = L₃(1), and for the second we consider V(1) to be the entire covariate history $\stackrel{‒}{L}\left(1\right)$. We have shown via Monte Carlo simulation that the optimal rule has mean outcome E_P0Y_d0 ≈ 0.536 when V(1) = L₃(1) and the optimal rule has mean outcome E_P0Y_d0 ≈ 0.563 when V(1) = (L₁(1), L₂(1), L₃(1), L₄(1)). One can verify that the blip function at the second time point is nonzero with probability 1 for both choices of V(1).

8.1.2 Two time point

We again simulate 1,000 data sets of 1,000 observations from an RCT without missingness. The observed variables have the following distribution:

$$L_1\left(0\right),L_2\left(0\right)\stackrel{iid}{~}\text{Unif}(-1,1)$$

$$A_1\left(0\right)\mid L\left(0\right)~\text{Bern}(1∕2)$$

$$A_2\left(0\right)\mid A_1\left(0\right),L\left(0\right)~\text{Bern}\left(1\right)$$

$$U_1,U_2\mid A\left(0\right),L\left(0\right)\stackrel{iid}{~}\text{Unif}(-1,1)$$

$$L_1\left(1\right)\mid A\left(0\right),L\left(0\right),U_1,U_2~U_1\left(1.25A_1\right(0)+0.25)$$

$$L_2\left(1\right)\mid A\left(0\right),L\left(0\right),L_1\left(1\right),U_1,U_2~U_2\left(1.25A_1\right(0)+0.25)$$

$$A_1\left(1\right)\mid A\left(0\right),\stackrel{‒}{L}\left(1\right)~\text{Bern}(1∕2)$$

$$A_2\left(1\right)\mid A\left(0\right),A_1\left(1\right),\stackrel{‒}{L}\left(1\right)~\text{Bern}\left(1\right)$$

$$Y\mid \stackrel{‒}{A}\left(1\right),\stackrel{‒}{L}\left(1\right)~\text{Bern}(0.4+0.069b(\stackrel{‒}{A}\left(1\right),\stackrel{‒}{L}\left(1\right)\left)\right),$$

where

$$b\left(\stackrel{‒}{A}\right(1),\stackrel{‒}{L}(1\left)\right)\equiv 0.5A_1\left(0\right)\left(-0.8-3\left(\text{sgn}\right(L_1\left(0\right))+L_1(0\left)\right)-L_2{\left(0\right)}^2\right)+A_1\left(1\right)\left(-0.35+{\left(L_1\right(1)-0.5)}^2\right)+0.08A_1\left(0\right)A_1\left(1\right).$$

Note that $E_{P_0}[Y\mid \stackrel{‒}{A}(1),\stackrel{‒}{L}(1\left)\right]$ is contained in the unit interval by the bounds on $\stackrel{‒}{A}\left(1\right)$ and $\stackrel{‒}{L}\left(1\right)$ so that Y is indeed a valid Bernoulli random variable. We will let V(0) = L(0) and $V\left(1\right)=\left(A\right(0),\stackrel{‒}{L}(1\left)\right)$. One can verify that (5) is satisfied for this choice of V.

Static treatments yield mean outcomes E_P0Y_(0,1),(0,1) = 0.400, E_P0Y_(0,1),(1,1) ≈ 0.395, E_P0Y_(1,1),(0,1) ≈ 0.361, and E_P0Y_(1,1),(1,1) ≈ 0.411. The true optimal treatment has mean outcome E_P0Y_d0 ≈ 0.485.

8.2 Optimal rule estimation methods

For now suppose we have estimators of the optimal rule with reasonable convergence properties, by which we mean that the true mean outcome under the fitted rule is close to the mean outcome under the optimal rule. In our companion paper in this volume we describe these estimators and show precisely how close these estimators come to achieving the optimal mean outcome. Here we note that our estimation algorithms correspond to using the full candidate library of weighted classification and blip function-based estimators proposed in table 2 of our companion paper, with the weighted log loss function used to determine the convex combination of candidates. We provide oracle inequalities for this estimator in our companion paper, and argue that it represents a powerful approach to data adaptively estimate the optimal rule without over- or underfitting the data. For a sample size n, we denote the rule estimated on the whole sample by d_n, and the rule estimated on training sample j by d_nj.

8.3 Inference procedures

We use four procedures to estimate the mean outcome under the fitted rule. All inference procedures rely on the intervention mechanism g₀. We always estimate the intervention mechanism with the true mechanism g₀, as one may do in an RCT without missingness. We do not consider efficiency gains resulting from estimating the known treatment mechanism here.

The first method uses the TMLE described in “TMLE of the mean outcome under a given rule” in Appendix B. The second method uses the analogous estimating equation approach that uses the double robust inverse probability of censoring weighted (DR-IPCW) estimating equation implied by $D^{\ast }(d_n,Q_n^{d_n},g_0)$, where $Q_n^{d_n}$ represents the unfluctuated initial estimates of $Q_0^{d_n}$. See van der Laan and Robins [34] for a general outline of such an estimating equation approach. This approach is valid whenever the TMLE is valid. We also use the CV-TMLE described in “CV-TMLE of the mean outcome under data adaptive V-optimal rule” in Appendix B, where we use a 10-fold cross-validation scheme. Finally, we use the CV-DR-IPCW cross-validated estimating equation implied by ${\Sigma }_j{P}_{n,j}^1{D}^{\ast }(d_{nj},Q_{nj}^{d_{nj}},g_0)$, where $Q_{nj}^{d_{nj}}$ represents the unfluctuated initial estimates of $Q_0^{d_{nj}}$. This approach is valid whenever the CV-TMLE is valid.

All inference procedures also rely on an estimate of $Q_0^d$ for some estimated d. For the two time point case, we use the empirical distribution of L(0) to estimate the marginal distribution of L(0). We compare plugging in both of the true values of $E_{P_0}[Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$ and E_P0 [Y_d|L(0, A(0) = d_A(0)(V(0)) as initial estimates with plugging in the incorrectly specified constant function 1/2 as initial estimates.

For the single time point case, we compare plugging in the true value of $E_{P_0}[Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$ with the incorrectly specified constant function 1/2. We always estimate E_P0 [Y_d|L(0), A(0) = d_A(0)(V(0))] by averaging

$$\left(A\right(0),\stackrel{‒}{L}(1)\mapsto E_{P_0}[Y\mid \stackrel{‒}{A}\left(1\right)=d\left(A\right(0),V),\stackrel{‒}{L}\left(1\right)]$$

over the empirical distribution of L(1) from the entire sample for non-cross-validated methods, and from the training sample for cross-validated methods. The empirical distribution of L(0) will not play a role for the single time point case because L(0) = ∅.

The procedures used to estimate the optimal rule rely on similar means, and we supply these estimation procedures with the incorrect value 1/2 for these conditional means whenever we supply the inference procedures with the incorrect values of the corresponding conditional means, and with the correct values of the conditional means whenever we supply the inference procedures with the corresponding correct values.

The simulation was implemented in R [57]. The code used to run the simulations is available upon request. We are currently looking to implement the methods in this paper and the companion paper in an R package.

8.4 Evaluating performance

We use the coverage of asymptotic 95% confidence intervals to evaluate the performance of the various methods. As we establish in the earlier parts of this paper, each inference approach yields two interesting target parameters with respect to which we can compute coverage. All approaches give asymptotically valid inference for the mean outcome under the optimal rule under conditions, and thus the coverage with respect to this parameter is assessed across all methods.

The TMLE and DR-IPCW estimating equation-based approaches also estimate the data adaptive target parameter ψ_0n as presented in Section 6. Given a fitted rule d_n, we approximate the expected value in this parameter definition using 10⁶ Monte Carlo simulations for the single time point case and 5 × 10⁵ Monte Carlo simulations for the two time point case. We then assess confidence interval coverage with respect to this approximation.

The CV-TMLE and cross-validated DR-IPCW estimating equation approaches estimate the data adaptive target parameter ${\tilde{\psi }}_{0n}$ as presented in Section 7. Given the ten rules estimated on each of the training sets, the expectation over the sample split random variable B_n becomes an average over ten target parameters, one for each estimated rule. Again we estimate the expected value of P₀ using 10⁶ Monte Carlo simulations for each of the ten target parameters in the single time point case, and 5 × 10⁵ Monte Carlo simulations in the two time point case.

9 Simulation results

Figure 1 shows that the (CV-)TMLE is more efficient than the (CV-)DR-IPCW estimating equation methods in our single time point simulation, except for the cross-validated methods when V = L₁(1),…, L₄(1) and the regressions are misspecified. Note that the MSEs relative to E_P0 Y_d0 are the typical E_P0(ψ_n − ψ₀)² for an estimate ψ_n, while the MSEs relative to the data adaptive parameter are the slightly less typical E_P0(ψ_n − ψ_0n)² for the TMLE and DR-IPCW, and $E_{P_0}{({\psi }_n-{\tilde{\psi }}_{0n})}^2$ for the cross-validated methods. That is, the target parameters vary for each of the 1,000 data sets considered. We also confirmed that, as is typical in missing data problems, the methods in which the conditional means were correctly specified were more efficient than the methods in which the conditional means are incorrectly specified. Figure 2 shows that the (CV-)TMLE in general has better coverage than the (CV-)DR-IPCW estimating equation approaches in our single time point simulation, with the only exception being the CV-TMLE for E_P0 Y_d0 when the regressions are misspecified and V = L₁(1),…, L₄(1).

Figure 1.

Relative efficiency of TMLE and DR-IPCW methods compared to both E_P0Y_d0 and the data adaptive parameter E_P0(ψ_n – ψ_0n)² for the TMLE and DR-IPCW, and $E_{P_0}{(\psi -{\tilde{\psi }}_{0n})}^2$ for the cross-validated methods. Results are provided both for the cases where the estimate $E_n[Y\mid \stackrel{‒}{A}(1),W]$ of is correctly specified and the case where this estimate is incorrectly specified with the constant function 1/2. Error bars indicate 95% confidence intervals to account for uncertainty from the finite number of Monte Carlo draws in our simulation. (a) V=L₁(1), (b) V=L₁(1), …, L₄(1).

Figure 2.

Coverage of 95% confidence intervals from the TMLE and DR-IPCW methods with respect to both E_P0 Y_d0 and the data adaptive parameter ψ_0n for the TMLE and DR-IPCW and ${\tilde{\psi }}_{0n}$ for the cross-validated methods. Results are provided both for the cases where the estimate $E_n[Y\mid \stackrel{‒}{A}(1),W]$ of $E_{P_0}[Y\mid \stackrel{‒}{A}(1),W]$ is correctly specified and the case where this estimate is incorrectly specified with the constant function 1/2. The (CV-)TMLE outperforms the (CV-)DR-IPCW estimating equation approach for almost all settings. Error bars indicate 95% confidence intervals to account for uncertainty from the finite number of Monte Carlo draws in our simulation. (a) V=L₁(1), (b) V=L₁(1), …, L₄(1).

Figure 3a shows that the (CV-)TMLE is always more efficient than the (CV-)DR-IPCW estimating equation methods for our two time point simulation. Figure 3b shows that this increased efficiency does not come at the expense of coverage: the (CV-)TMLE always has better coverage than the (CV-) DR-IPCW estimators in our two time point simulation. In general, we see that the cross-validated methods always achieve approximately 95% coverage for the data adaptive parameter. This is to be expected because the cross-validated methods only learn the optimal rule on validation sets, and thus avoid finite sample bias when the conditional means of the outcome are averaged over the validation samples.

Figure 3.

(a) Relative efficiency of TMLE and DR-IPCW methods compared to both E_P0 Y_d0 and the data adaptive parameter E_P0 (ψ_n – ψ_0n)² for the TMLE and DR-IPCW, and $E_{P_0}{({\psi }_n-{\tilde{\psi }}_{0n})}^2$ for the cross-validated methods. (b) Coverage of 95% confidence intervals from the TMLE and DR-IPCW methods with respect to both E_P0 Y_d0 and the data adaptive parameter ψ_0n for the TMLE and DR-IPCW and ${\tilde{\psi }}_{0n}$ for the cross-validated methods. Both (a) and (b) give results both for the cases where the estimates of $E_{P_0}[Y\mid \stackrel{‒}{A}(1)=d_n(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$ and E_P0[Y_dn|L(0)] are correctly specified and the case where these estimates are incorrectly specified with the constant function 1/2. Error bars indicate 95% confidence intervals to account for uncertainty from the finite number of Monte Carlo draws in our simulation.

It may at first be surprising that the TMLE outperforms the DR-IPCW estimating equation method in a randomized clinical trial, especially given that the CV-TMLE and CV-DR-IPCW achieve similar coverage. In Appendix C we give intuition as to why this may be the case in a single time point randomized clinical trial. In short, this difference in coverage appears to occur because our proposed TMLE only fluctuates the conditional means for individuals who received the fitted treatment, thereby reducing finite sample bias that may result from estimating the optimal rule on the same sample that is used to estimate the mean outcome under this fitted rule.

We also looked at the average confidence interval width across Monte Carlo simulations for each method and simulation setting. For a given simulation setting, all four estimation methods gave approximately the same (±0:002) average confidence interval width: 0.08 for both single time point simulations, 0.12 for the multiple time point simulation. These average widths show that we can get informatively small confidence intervals from our relatively small sample size of 1,000 individuals. Unlike Figures 1 and 3a, these values should not be used to gauge the efficiency of the proposed estimators since they do not take the true parameter value into account.

10 Discussion

This article investigated semiparametric statistical inference for the mean outcome under the V-optimal rule and statistical inference for the data adaptive target parameter defined as the mean outcome under a data adaptively determined V-optimal rule (treating the latter as given).

We proved a surprising and useful result stating that the mean outcome under the V-optimal rule is represented by a statistical parameter whose pathwise derivative is identical to what it would have been if the unknown rule had been treated as known, under the condition that the data is generated by a non-exceptional law [52]. As a consequence, the efficient influence curve is immediately known, and any of the efficient estimators for the mean outcome under a given rule can be applied at the estimated rule. In particular, we demonstrate a TMLE, and present asymptotic linearity results. However, the dependence of the statistical target parameter on the unknown rule affects the second-order terms of the TMLE, and, as a consequence, the asymptotic linearity of the TMLE requires that a second-order difference between the estimated rule and the V-optimal rule converges to zero at a rate faster than $1∕\sqrt{n}$. We show that this can be expected to hold for rules that are only a function of one continuous score (such as a biomarker), but when V is higher dimensional, only strong smoothness assumptions will guarantee this, so that, even in an RCT, we cannot be guaranteed valid statistical inference for such V-optimal rules.

Therefore, we proceeded to pursue statistical inference for so-called data adaptive target parameters. Specifically, we presented statistical inference for the mean outcome under the dynamic treatment regime we fitted based on the data. We showed that statistical inference for this data adaptive target parameter does not rely on the convergence rate of our estimated rule to the optimal rule, and in fact only requires that the data adaptively fitted rule converges to some (possibly suboptimal) fixed rule. However, even in a sequential RCT, the asymptotic linearity theorem still relies on an empirical process condition that limits the data adaptivity of the estimator of the rule. So, even though the assumptions are much weaker, they can still cause problems in finite samples when V is high dimensional, and possibly even asymptotically.

Therefore, we proceeded with the average of sample split specific target parameters, as in general proposed by van der Laan et al. [46], where we show that statistical inference can now avoid the empirical process condition. Specifically, our data adaptive target parameter is now defined as an average across J sample splits in training and validation sample of the mean outcome under the dynamic treatment fitted on the training sample. We presented CV-TMLE of this data adaptive target parameter, and we established an asymptotic linearity theorem that does not require that the estimated rule is consistent for the optimal rule, let alone at a particular rate. The CV-TMLE also does not require the empirical process condition. As a consequence, in a sequential RCT, this method provides valid asymptotic statistical inference without any conditions, beyond the requirement that the estimated rule converges to some (possibly suboptimal) fixed rule.

We supported our theoretical findings with simulations, both in the single and two time point settings. Our simulations supported our claim that it is easier to have good coverage of the proposed data adaptive target parameters than the mean outcome under the optimal rule, though the results for this harder mean outcome under the optimal rule parameter were also promising. In future work we hope to apply these methods to actual data sets of interest, generated by observational controlled trial as well as RCTs.

It might also be of interest to propose working models for the mean outcome E_P0 [Y_d0 |S] under the optimal rule, conditional on some baseline covariates S ⊂ W. This is now a function of S, but we would define the target parameter of interest as a projection of this true underlying function on the working model. It would now be of interest to develop TMLE for this finite dimensional pathwise differentiable parameter, and we presume that similar results as we found here might appear. Such parameters provide information about how the mean outcome under the optimal rule are affected by certain baseline characteristics.

Drawing inferences concerning optimal treatment strategies is an important topic that will hopefully help guide future health policy decisions. We believe that working with a large semiparametric model is desirable because it helps to ensure that the projected health benefits from implementing an estimated treatment strategy are not due to bias from a misspecified model. The TMLEs presented in this article have many desirable statistical properties and represent one way to get estimates and make inference in this large model. We look forward to future advances in statistical inference for parameters that involve optimal dynamic treatment regimes.

Appendix A

Proofs

Proof of Theorem 1. Let V_d = (V(0), V_d(1)). For a rule in $\mathcal{D}$, we have

$$\begin{array}{cc}\hfill E_{P_d}{Y}_d& =E_{P_d}{E}_{P_d}(Y_d\mid V_d)\hfill \\ \hfill & =E_{V_d}\left(E\right(Y_{a\left(0\right),a\left(1\right)}\mid V_{a\left(0\right)}\left)I\right(a\left(1\right)=d_{A\left(1\right)}\left(a\right(0),V_{a\left(0\right)}(1\left)\right)\left)I\right(a\left(0\right)=d_{A\left(0\right)}\left(V\right(0\left)\right)).\hfill \end{array}$$

For each value of a(0), V_a(0) = (V(0), V_a(0)(1)) and d_A(0)(V(0)), the inner conditional expectation is maximized over d_A(1)(a(0), V_a(0)(1)) by d_0,A(1) as presented in the theorem, where we used that V(1) includes V(0). This proves that d_0,A(1) is indeed the optimal rule for assignment of A(1). Suppose now that V(1) does not include V(0), but the stated assumption holds. Then the optimal rule d_0,A(1) that is restricted to be a function of (V(0), V(1), A(0)) is given by $\left({\stackrel{‒}{Q}}_{20}\right(A\left(0\right),V\left(0\right),V\left(1\right))>0)$, where

$${\stackrel{‒}{Q}}_{20}\left(a\right(0),v(0),v(1\left)\right)=E_{P_0}(Y_{a\left(0\right),A\left(1\right)=(1,1)}-Y_{a\left(0\right),A\left(1\right)=(0,1)}\mid V_{a\left(0\right)}(1)=v(1),V(0)=v(0\left)\right).$$

However, by assumption, the latter function only depends on (a(0), v(0), v(1)) through (a(0), v(1)), and equals ${\stackrel{‒}{Q}}_{20}\left(a\right(0),v(1\left)\right)$. Thus, we now still have that $d_{0,A\left(1\right)}\left(V\right)=\left(I\right({\stackrel{‒}{Q}}_{20}\left(A\right(0),V(1\left)\right)>0),1)$, and, in fact, it is now also an optimal rule among the larger class of rules that are allowed to use V(0) as well.

Given we found d_0,A(1), it remains to determine the rule d_0,A(0) that maximizes

$$E_{V_d}\left(E_P\left(Y_{a\left(0\right),d_{0,A\left(1\right)}}\mid V_{a\left(0\right)}\right)I\left(a\right(0)=d_{A\left(0\right)}(V\left(0\right)\left)\right)\right)=E_{P_0}E\left(Y_{a\left(0\right),d_{0,A\left(1\right)}}\mid V\left(0\right)\right)I\left(a\right(0)=d_{A\left(0\right)}(V\left(0\right)\left)\right),$$

where we used the iterative conditional expectation rule, taking the conditional expectation of V_a(0), given V(0). This last expression is maximized over d_A(0) by d_0,A(0) as presented in the theorem. This completes the proof.

The following lemma will be useful for proving Theorem 2.

Lemma 1. Recall the definitions of ${\stackrel{‒}{Q}}_{20}$ and ${\stackrel{‒}{Q}}_{10}$ in Theorem 1. We can represent Ψ(P₀) = E_P0 Y_d0 as follows:

$$\Psi \left(P_0\right)=E_{P_0}{Y}_{(0,1),(0,1)}+E_{P_0}\left[d_{0,A\left(1\right)}\right((0,1),V_{(0,1)}\left(1\right)\left){\stackrel{‒}{Q}}_{20}\right((0,1),V_{(0,1)}\left(1\right)\left)\right]+E_{P_0}{d}_{0,A\left(0\right)}\left(V\right(0\left)\right){\stackrel{‒}{Q}}_{10}\left(V\right(0\left)\right).$$

where V_(0,1)(1) is drawn under the G-computation distribution for which treatment (0, 1) is given at the first time point.

Proof of Lemma A.1. For a point treatment data structure O = (L(0), A(0), Y) and binary treatment A(0), we have for a rule $V\to d\left(V\right),E_{P_0}{Y}_d=E_{P_0}{Y}_0+E_{P_0}d\left(V\right){\stackrel{‒}{Q}}_0\left(V\right)$ with ${\stackrel{‒}{Q}}_0\left(V\right)=E_{P_0}[Y_1-Y_0\mid V]$. This identity is applied twice in the following derivation:

$$\begin{array}{cc}\hfill \Psi \left(P_0\right)& =E_{P_0}{Y}_{(0,1),d_{0,A\left(1\right)}}+E_{P_0}{d}_{0,A\left(0\right)}\left(V\right(0\left)\right){\stackrel{‒}{Q}}_{10}\left(V\right(0\left)\right)\hfill \\ \hfill & =E_{P_0}{E}_{P_0}\left[Y_{(0,1),d_{0,A\left(1\right)}}\mid V_{(0,1)}\left(1\right)\right]+E_{P_0}{d}_{0,A\left(0\right)}\left(V\right(0\left)\right){\stackrel{‒}{Q}}_{10}\left(V\right(0\left)\right)\hfill \\ \hfill & =E_{P_0}{E}_{P_0}[Y_{(0,1),(0,1)}\mid V_{(0,1)}(1\left)\right]+E_{P_0}I\left({\stackrel{‒}{Q}}_{20}\right((0,1),V_{(0,1)}\left(1\right))>0){\stackrel{‒}{Q}}_{20}(0,V_{(0,1)}(1\left)\right)+E_{P_0}{d}_{0,A\left(0\right)}\left(V\right(0\left)\right){\stackrel{‒}{Q}}_{10}\left(V\right(0\left)\right)\hfill \\ \hfill & =E_{P_0}{E}_{P_0}[Y_{(0,1),(0,1)}\mid V_{(0,1)}(1\left)\right]+E_{P_0}{d}_{0,A\left(1\right)}\left(\right(0,1),V_{(0,1)}(1\left)\right){\stackrel{‒}{Q}}_{20}(0,V_{(0,1)}(1\left)\right)+E_{P_0}{d}_{0,A\left(0\right)}\left(V\right(0\left)\right){\stackrel{‒}{Q}}_{10}\left(V\right(0\left)\right)\hfill \\ \hfill & =E_{P_0}{Y}_{(0,1),(0,1)}+E_{P_0}{d}_{0,A\left(1\right)}\left(\right(0,1),V_{(0,1)}(1\left)\right){\stackrel{‒}{Q}}_{20}(0,V_{(0,1)}(1\left)\right)+E_{P_0}{d}_{0,A\left(0\right)}\left(V\right(0\left)\right){\stackrel{‒}{Q}}_{10}\left(V\right(0\left)\right).\hfill \end{array}$$

Proof of Theorem 3. By the definition of R_1d we have

$$\begin{array}{cc}\hfill P_0& D^{\ast }(Q,g)=P_0{D}^{\ast }(d_Q,Q,g)={\Psi }_{d_Q}\left(Q_0^{d_Q}\right)-{\Psi }_{d_Q}\left(Q^{d_Q}\right)+R_{1d_Q}(Q^{d_Q},Q_0^{d_Q},g,g_0)\hfill \\ \hfill & ={\Psi }_{d_0}\left(Q_0^{d_0}\right)-{\Psi }_{d_Q}\left(Q^{d_Q}\right)+\left\{{\Psi }_{d_Q}\right(Q_0^{d_Q})-{\Psi }_{d_0}(Q_0^{d_0}\left)\right\}+R_{1d_Q}(Q^{d_Q},Q_0^{d_Q},g,g_0)\hfill \\ \hfill & =\Psi \left(Q_0\right)-\Psi \left(Q\right)+R_2(Q,Q_0)+R_{1d_Q}(Q^{d_Q},Q_0^{d_Q},g,g_0).\hfill \end{array}$$

Proof of Lemma 1. Below we omit the dependence of d_Q,A(0), d_0,A(0), ${\stackrel{‒}{Q}}_1$, and ${\stackrel{‒}{Q}}_{10}$ on V(0):

$$\begin{array}{cc}\hfill R_{2A\left(0\right)}& =E_{P_0}\left[\right(d_{Q,A\left(0\right)}-d_{0,A\left(0\right)}\left){\stackrel{‒}{Q}}_{10}\right]\hfill \\ \hfill & \le E_{P_0}\mid (d_{Q,A\left(0\right)}-d_{0,A\left(0\right)}){\stackrel{‒}{Q}}_{10}\mid \hfill \\ \hfill & =E_{P_0}\mid (d_{Q,A\left(0\right)}-d_{0,A\left(0\right)}){\stackrel{‒}{Q}}_{10}I(\mid {\stackrel{‒}{Q}}_{10}\mid \ge \mid {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}\mid )\mid +E_{P_0}\mid (d_{Q,A\left(0\right)}-d_{0,A\left(0\right)}){\stackrel{‒}{Q}}_{10}I(0<\mid {\stackrel{‒}{Q}}_{10}\mid <\mid {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}\mid )\mid .\hfill \end{array}$$

The first term in the final equality is always 0 because d_Q,A(0) = d_0,A(0) whenever the indicator is 1. In the second term, d_Q,A(0) ≠ d_0,A(0) whenever the indicator is 1, so:

$$\begin{array}{cc}\hfill R_{2A\left(0\right)}& \le E_{P_0}[\mid {\stackrel{‒}{Q}}_{10}\mid I(0<\mid {\stackrel{‒}{Q}}_{10}\mid <\mid {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}\mid \left)\right]\hfill \\ \hfill & \le E_{P_0}\left[\mid {\stackrel{‒}{Q}}_{10}\mid I\left(0<\mid {\stackrel{‒}{Q}}_{10}{\mid }^{\frac{p({\beta }_1+1)}{p+{\beta }_1}}<\mid {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}{\mid }^{\frac{p({\beta }_1+1)}{p+{\beta }_1}}\right)I(\mid {\stackrel{‒}{Q}}_{10}\mid >0)\right]\hfill \\ \hfill & \le E_{P_0}\left[\mid {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}{\mid }^{\frac{p({\beta }_1+1)}{p+{\beta }_1}}\mid {\stackrel{‒}{Q}}_{10}{\mid }^{-\frac{{\beta }_1(p-1)}{p+{\beta }_1}}I(\mid {\stackrel{‒}{Q}}_{10}\mid >0)\right]\hfill \\ \hfill & \le \Vert {\stackrel{‒}{Q}}_1-{\stackrel{‒}{Q}}_{10}{\Vert }_{p,P_0}^{\frac{p({\beta }_1+1)}{p+{\beta }_1}}\Vert {\stackrel{‒}{Q}}_{10}^{-1}I(\mid {\stackrel{‒}{Q}}_{10}\mid >0){\Vert }_{{\beta }_1,P_0}^{\frac{{\beta }_1(p-1)}{p+{\beta }_1}}\hfill \end{array}$$ (15)

where the final inequality holds by Hölder's inequality. The above also holds when the limit is taken as p ≠ ∞, yielding the essential supremum result. The result for R_2A(1) follows by the same argument.

Proof of Theorem 4. By Theorem 3, we have

$$P_0{D}^{\ast }(d_n,Q_n^{d_n\ast },g_n)={\psi }_0-{\Psi }_{d_n}\left(Q_n^{d_n\ast }\right)+R_n,$$

where $R_n=R_{1d_n}(Q_n^{d_n},Q_0^{d_n},g_n,g_0)+R_2(Q_n,Q_0)$. Combining this with the fact that $D_n^{\ast }\equiv D^{\ast }(d_n,Q_n^{d_n\ast },g_n)$ has empirical mean 0 yields

$${\Psi }_n^{\ast }-{\Psi }_0=(P_n-P_0)D_n^{\ast }+R_n=(P_n-P_0)D^{\ast }(d_0,Q^{d_0},g_0)+(P_n-P_0)(D_n^{\ast }-D^{\ast }(d_0,Q^{d_0},g_0\left)\right)+R_n$$

The Donsker condition and the mean square consistency of $D_n^{\ast }\text{to}{D}^{\ast }(d_0,Q^{d_0},g_0)$ give

$$(P_n-P_0)(D_n^{\ast }-D^{\ast }(d_0,Q^{d_0},g_0\left)\right)=o_{P_0}\left(n^{-1∕2}\right),$$

see, for example, van der Vaart and Wellner [58]. By assumption, R₂(Q_n, Q₀) = o_P0 (n^−1/2). Thus:

$${\Psi }_n^{\ast }-{\Psi }_0=(P_n-P_0)D^{\ast }(d_0,Q^{d_0},g_0)+R_{1d_n}(Q_n^{d_n},Q_n^{d_n},g_n,g_0)+o_{P_0}\left(n^{-1∕2}\right)$$

as desired.

Proof of Theorem 6. For all j = 1,…, J, we have that:

$${\Psi }_{d_{nj}}\left(Q_{nj}^{d_{n{j}^{\ast }}}\right)-{\Psi }_{d_{nj}}\left(Q_0^{d_{n{j}^{\ast }}}\right)=-P_0{D}^{\ast }(d_{nj},Q_{nj}^{d_{n{j}^{\ast }}},g_{nj})+R_{1d_{nj}}(Q_{nj}^{d_{n{j}^{\ast }}},Q_0^{d_{n{j}^{\ast }}},g_{nj},g_0)$$

Summing over j and using (13) gives:

$${\tilde{\Psi }}_n^{\ast }-{\tilde{\Psi }}_{0n}=\frac{1}{J}\sum _{j=1}^J\left(\right(P_{n,j}^1-P_0\left)D^{\ast }\right(d_{nj},Q_{nj}^{d_{n{j}^{\ast }}},g_{nj})+R_{1d_{nj}}(Q_{nj}^{d_{n{j}^{\ast }}},Q_0^{d_{n{j}^{\ast }}},g_{nj},g_0\left)\right).$$

We also have that:

$$\frac{1}{J}\sum _{j=1}^J(P_{n,j}^1-P_0)\left(D^{\ast }(d_{nj},Q_{nj}^{d_{n{j}^{\ast }}},g_{nj})-D^{\ast }(d_1,Q^{d_1},g)\right)=o_{P_0}\left(n^{-1∕2}\right).$$

The above follows from the first by applying the law of total expectation conditional on the training sample, and then noting that each ${\widehat{Q}}^{\ast }(P_{n,B_n}^0,{\epsilon }_n)$ only relies on $P_{n,B_n}^0$ through the finite dimensional parameter ε_n. Because GLM-based parametric classes easily satisfy an entropy integral condition [58], the consistency assumption on $D^{\ast }(d_{nj},Q_{nj}^{d_{nj}\ast },g_{nj})$ shows that the above is second order. We refer the reader to Zheng and van der Laan [55] for a detailed proof of the above result for general cross-validation schemes, including J-fold cross-validation.

It follows that:

$${\tilde{\Psi }}_n^{\ast }-{\tilde{\Psi }}_{0n}=(P_n-P_0)D^{\ast }(d_1,Q^{d_1},g)+\frac{1}{J}\sum _{j=1}^J{R}_{1d_{nj}}\left(Q_{nj}^{d_{n{j}^{\ast }}},Q_0^{d_{n{j}^{\ast }}},g_{nj},g_0\right)+o_{P_0}\left(n^{-1∕2}\right).$$

Appendix B: Estimators of the mean outcome under the optimal rule

TMLE of the mean outcome under a given rule

This TMLE for a fixed dynamic treatment rule has been presented in the literature, but for the sake of being self-contained it will be shortly described here. The TMLE yields a substitution estimator that empirically solves the estimating equations corresponding to the efficient influence curve, analogous to Theorem 2 for general d. By substitution estimator, we mean that the TMLE can be written as the mapping Ψ applied to a particular Q.

Assume without loss of generality that Y ∈ [0, 1]. In this section we use lower case letters to emphasize when quantities are the values taken on by random variables rather than the random variables themselves, for example, our sample is given by (o₁; …, o_n), where $o_i=(\stackrel{‒}{l}{\left(1\right)}_i,\stackrel{‒}{a}{\left(1\right)}_i,y_i)$. The indicator for not being right censored at time j for individual i is given by a₂(j)_i.

Regress (y_i : a₂(0)_i = a₂(1)_i = 1) on $(\stackrel{‒}{a}{\left(1\right)}_i,\stackrel{‒}{l}{\left(1\right)}_i:a_2{\left(0\right)}_i=a_2{\left(1\right)}_i=1)$ to get an estimate

$$\left(a_1\right(0),a_1(1),\stackrel{‒}{l}(1\left)\right)\mapsto E_n[Y\mid \stackrel{‒}{A}(1)=(\left(a_1\right(0),1)\left(a_1\right(1),1)),\stackrel{‒}{L}(1)=\stackrel{‒}{l}(1\left)\right].$$ (16)

Note that we have only used individuals who are not right censored at time 1 to obtain this fit. The above regression can be fitted using a data adaptive technique such as super-learning [59]. To estimate ${E_P}_0[Y\mid \stackrel{‒}{A}(1)=d(a\left(0\right),v),\stackrel{‒}{l}(1\left)\right]$, use

$$\left(a\right(0),\stackrel{‒}{l}(1\left)\right)\mapsto E_n[Y\mid \stackrel{‒}{A}(1)=d(a\left(0\right),v),\stackrel{‒}{L}(1)=\stackrel{‒}{l}(1\left)\right],$$

where we remind the reader that we are treating the rule d = d_n as a known function and that v is a function of $\stackrel{‒}{l}\left(1\right)$ that sets the indicators for not being censored to 1. Consider the fluctuation submodel

$$\text{logit}{E}_n^{\left({\epsilon }_2\right)}[Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]=\text{logit}{E}_n[Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]+{\epsilon }_2{H}_2\left(g_n\right)\left(O\right),$$

where

$$H_2\left(g_n\right)\left(O\right)=\frac{I\left(\stackrel{‒}{A}\right(1)=d(A\left(0\right),V\left)\right)}{\prod _{j=1}^1{g}_{n,A\left(j\right)}\left(O\right)}.$$

Let ε_2n be the estimate for ε₂ obtained by running a univariate logistic regression of (y_i : i = 1, …, n) on (H₂(g_n)(o_i) : i = 1, … n) using

$$\left(\text{logit}{E}_n\right[Y\mid \stackrel{‒}{A}\left(1\right)=d(a{\left(0\right)}_i,v_i),\stackrel{‒}{L}\left(1\right)=\stackrel{‒}{l}{\left(1\right)}_i]:i=1,\dots ,n)$$

as offset. This defines a targeted estimate

$$E_n^{\ast }[Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]\equiv E_n^{\left({\epsilon }_{2n}\right)}[Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$$ (17)

of the regression function, where we remind the reader that the targeted estimate is chosen to ensure that the empirical mean of the component $D_2^{\ast }$ is 0 when we plug in the estimate of the intervention mechanism and the targeted estimate of the regression function for the unknown true quantities.

We now develop a targeted estimator of the second regression function in $D_1^{\ast }$ to ensure that the substitution estimator of $D_1^{\ast }$ will have empirical mean 0. Regress

$$\left(E_n\right[Y\mid \stackrel{‒}{A}\left(1\right)=d(a{\left(0\right)}_i,v_i),\stackrel{‒}{L}\left(1\right)=\stackrel{‒}{l}{\left(1\right)}_i]:a_2{\left(0\right)}_i=1)$$

on (l(0)_i; a(0)_i : a₂(0)_i = 1) to get the regression function

$$\left(a_1\right(0),l(0\left)\right)\mapsto E_n\left[E_n\right[Y\mid \stackrel{‒}{A}\left(1\right)=d\left(A\right(0),V),\stackrel{‒}{L}\left(1\right)]\mid A(0)=(a_1\left(0\right),1),L(0)=l(0\left)\right].$$ (18)

One can estimate this quantity using the super-learner algorithm among all individuals who are not right censored at time 0. For honest cross-validation in the super-learner algorithm, the nuisance parameter $E_n[Y\mid \stackrel{‒}{A}(1)=d(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$ should be fit on the training samples in the super-learner algorithm. We refer the reader to Appendix B of van der Laan and Gruber [41] for a detailed explanation of this procedure. The same strategy holds for estimating the nuisance parameter g₀ when necessary (e.g., in an observational study).

For an estimate of E_P0[Y_d|L(0)], one can use the regression function above, but with a(0) fixed to d_A(0)(v(0)), which is itself a function of l(0). We will denote this function by l(0) ↦ E_n[Y_d|L(0) = l(0)]. We now wish to fluctuate this initial estimator so that the plug-in estimator of $D_1^{\ast }\left(P_0\right)$ has empirical mean 0. In particular, we use the submodel

$$\text{logit}{E}_n^{\left({\epsilon }_1\right)}[Y_d\mid L(0\left)\right]=\text{logit}{E}_n[Y_d\mid L(0\left)\right]+{\epsilon }_1{H}_1\left(g_n\right),$$

where

$$H_1\left(g_n\right)=\frac{I\left(A\left(0\right)=d_{A\left(0\right)}\left(V\right(0\left)\right)\right)}{g_{n,A\left(0\right)}\left(O\right)}.$$

Let ε_1n be the estimate for ε₁ obtained by running a univariate logistic regression of

$$\left(E_n^{\ast }\right[Y\mid \stackrel{‒}{A}\left(1\right)=d(a{\left(0\right)}_i,v_i),\stackrel{‒}{L}\left(1\right)=\stackrel{‒}{l}{\left(1\right)}_i]:i=1,\dots ,n)$$

on (H₁(g_n)(o_i) : i = 1, …, n) using (logitE_n[Y_d|L(0) = l(0)_i : i = 1, …, n) as offset. A targeted estimate of E_P0[Y_d|L(0) is given by

$$E_n^{\ast }[Y_d\mid L(0)\equiv E_n^{\left({\epsilon }_{1n}\right)}[Y_d\mid L\left(0\right)]$$ (19)

Plugging the targeted regressions and g_n into the expression for $D_1^{\ast }$ shows that this estimate of $D_1^{\ast }$ has empirical mean 0.

Let Q_L(0);n be the empirical distribution of L(0), and let $Q_n^{d\ast }$ be the parameter mapping representing the collection containing Q_L(0),n and the targeted regression functions in (17) and (19). This concludes the presentation of the components of the TMLE of E_P0Y_d. The discussion of properties of this estimator is continued in the main text.

CV-TMLE of the mean outcome under data adaptive V-optimal rule

Let $\widehat{d}:\mathcal{M}\to \mathcal{D}$ be an estimator of the V-optimal rule d₀. Firstly, without loss of generality we can assume that Y ∈ [0, 1]. Denote the realizations of B_n with j = 1, …, J, and let $d_{nj}=\widehat{d}\left(P_{n,j}^0\right)$ denote the estimated rule on training sample j. Let

$$\left(a\right(0),\stackrel{‒}{l}(1\left)\right)\mapsto E_{nj}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(a\left(0\right),v),\stackrel{‒}{L}(1)=\stackrel{‒}{l}(1\left)\right]$$ (20)

represent an initial estimate of $E_{P_0}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$ based on the training sample j, obtained analogously to the estimator in (16). Similarly, let g_nj represent the estimated intervention mechanism based on this training sample $P_{n,j}^0$, j = 1, …, J. Consider the fluctuation submodel

$$\text{logit}{E}_{nj}^{\left({\epsilon }_2\right)}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]=\text{logit}{E}_{nj}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]+{\epsilon }_2{H}_2\left(g_{nj}\right)\left(O\right)$$

where

$$H_2\left(g_{nj}\right)\left(O\right)=\frac{I\left(\stackrel{‒}{A}\left(1\right)=d_{nj}\left(A\right(0),V(1\left)\right))\right)}{\prod _{l=0}^1{g}_{nj,A\left(l\right)}\left(O\right)}.$$

Note that the fluctuation ε₂ does not rely on j. Let

$${\epsilon }_{2n}=\text{arg}\underset{{\epsilon }_2}{\text{min}}\frac{1}{J}\sum _{j=1}^J{P}_{n,j}^1\tilde{\varphi }\left(E_{nj}^{\left({\epsilon }_2\right)}\right),$$

where $E_{nj}^{\left({\epsilon }_2\right)}$ represents the fluctuated function in (20) and

$$-\tilde{\varphi }\left(f\right)\left(o\right)=y\log f\left(o\right)+(1-y)\log (1-f(o\left)\right).$$ (21)

for all $f:\mathcal{O}\to (0,1)$. For each i = 1, …, n, let j(i) ∈ {1; …, J} represent the value of B_n for which element i is in the validation set. The fluctuation ε_2n can be obtained by fitting a univariate logistic regression of (y_i: i = 1, …, n) on (H₂(g_nj(i))(o_i): i = 1, …, n) using

$$\left(\text{logit}{E}_{nj\left(i\right)}\left[Y\mid \stackrel{‒}{A}\left(1\right)=d_{nj}(a{\left(0\right)}_i,v_i),\stackrel{‒}{L}\left(1\right)=\stackrel{‒}{l}{\left(1\right)}_i\right]:i=1,\dots ,n\right)$$

as offset. Thus each observation i is paired with nuisance parameters that are fit on the training sample which does not contain observation i. This defines a targeted estimate

$$E_{nj}^{\ast }\left[Y\mid \stackrel{‒}{A}\left(1\right)=d_{nj}\left(A\right(0),V),\stackrel{‒}{L}\left(1\right)\right]\equiv E_{nj}^{\left({\epsilon }_{2n}\right)}\left[Y\mid \stackrel{‒}{A}\left(1\right)=d_{nj}\left(A\right(0),V),\stackrel{‒}{L}\left(1\right)\right]$$ (22)

of $E_{P_0}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$. We note that this targeted estimate only depends on P_n through the training sample $P_{n,j}^0$ and the one-dimensional ε_2n.

We now aim to get a targeted estimate of E_P0[Y_dnj|L(0)]. We can obtain an estimate

$$\left(a_1\right(0),l(0\left)\right)\mapsto E_{nj}\left[E_{nj}\left[Y\mid \stackrel{‒}{A}\left(1\right)=d_{nj}\left(A\right(0),V),\stackrel{‒}{L}\left(1\right)\right]\mid A\left(0\right)=\left(a_1\right(0),1),L\left(0\right)=l\left(0\right)\right]$$ (23)

in the same manner as we estimated the quantity in (18), with the caveat that we replace $E_n[Y\mid \stackrel{‒}{A}(1)=d_{nj}(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$ by $E_{nj}[Y\mid \stackrel{‒}{A}(1)=d_{nj}(A\left(0\right),V),\stackrel{‒}{L}(1\left)\right]$ and only fit the regression on samples that are not right censored at time 0 and are in training set j. For an estimate E_nj[Y_dnj|L(0)] of E_P0[Y_dnj|L(0)], we can use the regression function above but with a(0) fixed to d_nj,A(0)(v(0)).

Consider the fluctuation submodel

$$\text{logit}{E}_{nj}^{\left({\epsilon }_1\right)}\left[Y_{d_{nj}}\mid L\left(0\right)\right]=\text{logit}{E}_{nj}\left[Y_{d_{nj}}\mid L\left(0\right)\right]+\epsilon H_1\left(g_{nj}\right)\left(O\right),$$

where

$$H_1\left(g_{nj}\right)\left(O\right)=\frac{I\left(A\right(0)=d_{nj,A\left(0\right)}(V\left(0\right)\left)\right)}{g_{nj,A\left(0\right)}\left(O\right)}.$$

Again the fluctuation ε₁ does not rely on j. Let

$${\epsilon }_{1n}=\text{arg}\underset{{\epsilon }_1}{\text{min}}\frac{1}{J}\sum _{j=1}^J{P}_{n,j}^1\tilde{\varphi }\left(E_{nj}^{\left({\epsilon }_1\right)}\right),$$

where $\tilde{\varphi }$ is defined in (21). For each i = 1, …, n, again let j(i) ∈ {1, …, J} represent the value of B_n for which element i is in the validation set. The fluctuation ε_1n can be obtained by fitting a univariate logistic regression of

$$\left(E_{nj\left(i\right)}^{\ast }\left[Y\mid \stackrel{‒}{A}\left(1\right)=d_{nj\left(i\right)}(a{\left(0\right)}_i,v_i),\stackrel{‒}{l}{\left(1\right)}_i\right]:i=1,\dots ,n\right)$$

on (H₁(g_nj(i))(o_i): i = 1, …, n) using

$$\left(\text{logit}{E}_{nj\left(i\right)}\left[Y_{d_{nj\left(i\right)}}\mid L\left(0\right)=l{\left(0\right)}_i\right]:i=1,\dots ,n\right)$$

as offset. This defines a targeted estimate

$$E_{nj}^{\ast }\left[Y_{d_{nj}}\mid L\left(0\right)\right]\equiv E_{nj}^{\left({\epsilon }_{1n}\right)}\left[Y_{d_{nj}}\mid L\left(0\right)\right]$$ (24)

of E_P0[Y_dnj|L(0)]. We note that this targeted estimate only depends on P_n through the training sample $P_{n,j}^0$ and the one-dimensional ε_1n.

Let Q_L(0),nj be the empirical distribution of L(0)_i for the validation sample $P_{n,j}^1$. For all j = 1, …, J, let $Q_{nj}^{d_{nj}\ast }$ be the parameter mapping representing the collection containing Q_L(0),nj and the targeted regressions in (22) and (24). This defines an estimator ${\psi }_{nj}^{\ast }=P_{n,j}^1{\stackrel{‒}{Q}}_{1nj}^{\ast }$ of ψ_dnj0 = Ψ_dnj(P₀) for each j = 1, …, J. CV-TMLE is now defined as ${\psi }_n^{\ast }=\frac{1}{J}{\Sigma }_{j=1}^J{\psi }_{nj}^{\ast }$. This CV-TMLE solves the cross-validated efficient influence curve equation:

$$\frac{1}{J}\sum _{j=1}^J{P}_{n,j}^1{D}^{\ast }\left(d_{nj},Q_{nj}^{d_{n{j}^{\ast }}},g_{nj}\right)=0.$$

Further, each $Q_{nj}^{d_{nj}\ast }$ only relies on $P_{n,j}^1$ through the univariate parameters ε_1n and ε_2n. This will allow us to use the entropy integral arguments presented in Zheng and van der Laan [55] which show that no restrictive empirical process conditions are needed on the initial estimates in (20) and (23).

The only modification relative to the original CV-TMLE presented in Zheng and van der Laan [55] is that in the above description we change our target on each training sample into the training sample-specific target parameter implied by the fit $\widehat{d}\left(P_{n,B_n}^0\right)$ on the training sample, while in the original CV-TMLE formulation, the target would still be Ψ_d0(P₀). With this minor twist, the (same) CV-TMLE is now used to target the average of training sample-specific target parameters averaged across the J training samples. This utilization of CV-TMLE was already used to estimate the average (across training samples) of the true risk of an estimator based on a training sample in van der Laan and Petersen [53] and Díaz and van der Laan [54], so that this just represents a generalization of that application of CV-TMLE to estimate general data adaptive target parameters as proposed in van der Laan et al. [46].

Appendix C: Why the TMLE may have better coverage than the estimating equation approach in a randomized clinical trial

We wrote this section after performing our simulations because we wanted to understand why the TMLE is outperforming the DR-IPCW estimating equation approach by such a wide margin. The two approaches do not typically give such disparate estimates in a randomized clinical trial, so it is natural to ask why this is happening in our simulations. Part of this section is conjecture (which is in line with our simulations), but we offer some justification to support this conjecture.

We now offer a heuristic explanation of why the TMLE may have better coverage than the DR-IPCW estimating equation approach when estimating the data adaptive parameter ψ_0n. Suppose we have a single time point data structure O = (W, A, Y) drawn according to the distribution P₀ in a randomized clinical trial without missingness. Here we use notation which directly describes the single time point data structure rather than forcing this problem into the longitudinal context as in Section 8.1.1. Let d₀ = arg max_d E_P0E_P0[Y|A = d(V), W] for some V that is a function of W. Suppose we observe o₁, …, o_n and let d_n be an estimate of d₀, which is obtained using the methods in our accompanying technical report [47]. For any fixed rule d, the efficient influence curve at some $P\in \mathcal{M}$ is given by

$$E_P\left[\frac{I(A=d(V\left)\right)}{g(A\mid W)}(Y-E_P[Y\mid A=d\left(V\right),W\left]\right)\right]+E_P[Y\mid A=d(V),W]-E_P{E}_P[Y\mid A=d(V),W],$$

where g is the intervention mechanism under P. Again we have that E_P0Y_d0 has the same influence curve as above with d = d₀ (see our online technical report). Suppose that g₀ = 1/2 is known and we have estimated E_P0 [Y|A = d(V), W] perfectly, though we continue to work in the model where E_P0 [Y|A = d(V), W] is treated as unknown so that simply averaging over this quantity is not appropriate if we want inference or robustness.

For any fixed rule V ↦ d(V), it is easy to show that

$$E_{P_0}\left[\frac{I(A=d(V\left)\right)}{g_0(A\mid W)}(Y-E_{P_0}[Y\mid A=d\left(V\right),W\left]\right)\right]=0,$$

where g₀(a|w) represents the probability under P₀ that A = a given W = w. Similarly, we expect that

$${\beta }_d\left(P_n\right)\equiv \frac{1}{n}\sum _{i=1}^n\frac{I(a_i=d(v_i\left)\right)}{g_0(a_i\mid w_i)}(y_i-E_{P_0}[Y\mid A=d\left(v_i\right),W=w_i\left]\right)\approx 0.$$

Further, E_P0β_d(P_n) = 0 for fixed d, where the expectation is over the observed sample P_n but not the fixed rule d. In the first part of this paper we argued that one can learn an estimated rule d_n on the entire data set, and then treat this rule d_n as known when estimating E_P0Y_dn. This is asymptotically valid under the conditions given in this paper, but even if these conditions hold we may expect some finite sample bias. In our simulation this finite sample bias is manifested as

$$E_{P_0}\left[\frac{1}{n}\sum _{i=1}^n\frac{I(a_i=d_n(v_i\left)\right)}{g_0(a_i\mid w_i)}(y_i-E_{P_0}[Y\mid A=d_n\left(v_i\right),W=w_i\left]\right)\right]>0,$$

where the expectation is over the observed sample P_n and the estimated rule d_n. For a single time point simulation with V = L₃(1), this sample average is approximately 0.013 on average across 1,000 simulations. When V = L₁(1), …, L₁(4), this sample average is approximately 0.040 on average across 1,000 simulations. Because this was a follow-up analysis, we ran these simulations on different Monte Carlo draws than those used for our results in the main text. We conjecture that the above phenomenon is not specific to our simulation settings and will occur in more general settings. Our companion paper in this issue explores the estimation of d₀, and a careful look at the mean performance-based loss function presented in that paper will show that indeed one way to make the empirical risk smaller is to choose d_n so that β_dn (P_n) > 0. Nonetheless, selecting d_n by a cross-validation selector as we propose in our companion paper should help mitigate this issue since β_dn for d_n trained on a training sample should have empirical mean close to 0 in the validation sample.

The DR-IPCW estimating equation gives the estimator:

$${\widehat{\Psi }}_{EE}^{d_n}\left(P_n\right)\equiv {\psi }_{n,EE}\equiv {\beta }_{d_n}\left(P_n\right)+\frac{1}{n}\sum _{i=1}^n{E}_{P_0}[Y\mid A=d_n(V_i),W=W_i].$$

This estimator has bias E_P0β_dn (P_n), where the expectation is over the random sample P_n and the estimated rule d_n.

Consider the simple linear TMLE which fluctuates w ↦ E_P0 [Y|A = d_n(v), W = w] using the submodel:

$$E_{P_0}^{\left(\epsilon \right)}[Y\mid A=d_n(V),W]=E_{P_0}[Y\mid A=d_n(V),W]+\epsilon \frac{I(A=d_n(V\left)\right)}{g_0(A\mid W)}$$

where we recall that w ↦ E_P0 [Y|A = d_n(v), W = v] is being treated as unknown. A valid TMLE is given by choosing ε_n to minimize the mean-squared error between Y and $E_{P_0}^{\left(\epsilon \right)}[Y\mid A=d_n(V),W]$. When Y is bounded, the logistic fluctuations that we have presented in this paper are preferable to the linear fluctuation because they respect our model constraints. We consider the linear fluctuation here for simplicity. The minimizer ε_n is given by

$$\begin{array}{cc}\hfill {\epsilon }_n& =\frac{\frac{1}{n}\sum _i\frac{I(a_i=d_n(v_i\left)\right)}{g_0(a_i\mid w_i)}(y_i-E_{P_0}[Y\mid A=d_n\left(v_i\right),W=w_i\left]\right)}{\frac{1}{n}\sum _i\frac{I(a_i=d_n(v_i\left)\right)}{g_0{(a_i\mid w_i)}^2}}\hfill \\ \hfill & =\frac{1}{2}\frac{{\beta }_{d_n}\left(P_n\right)}{\frac{1}{n}\sum _i\frac{I(a_i=d_n(v_i\left)\right)}{g_0(a_i\mid w_i)}},\hfill \end{array}$$

if $\frac{1}{n}{\Sigma }_i\frac{I(a_i=d_n(v_i\left)\right)}{g_0(a_i\mid w_i)}>0$ and we take $\frac{1}{n}{\Sigma }_i\frac{I(a_i=d_n(v_i\left)\right)}{g_0(a_i\mid w_i)}=0$. The denominator above is the same as the denominator in a modified Horvitz-Thompson estimator [60] and, more importantly, appears in one of the terms in the TMLE, which is given by

$$\begin{array}{cc}\hfill {\psi }_{n,\text{TMLE}}^{\ast }& \equiv \frac{1}{n}\sum _{i=1}^n{E}_{P_0}^{\left({\epsilon }_n\right)}[Y\mid A=d_n(V),W]\hfill \\ \hfill & =\frac{1}{n}\sum _{i=1}^n{E}_{P_0}[Y\mid A=d_n(V),W]+\frac{{\epsilon }_n}{n}\sum _{i=1}^n\frac{I(A=d_n(V\left)\right)}{g_0(A\mid W)}\hfill \\ \hfill & =\frac{1}{n}\sum _{i=1}^n{E}_{P_0}[Y\mid A=d_n(V),W]+\frac{{\beta }_{d_n}\left(P_n\right)}{2}.\hfill \end{array}$$

This linear fluctuation TMLE has bias $E_{P_0}\left[\frac{{\beta }_{d_n}\left(P_n\right)}{2}\right]$, which is half the bias of ${\widehat{\Psi }}_{EE}^{d_n}\left(P_n\right)$.

The arguments presented in this section are mainly interesting if E_P0 [β_dn (P_n) ≠ 0. We have conjectured that E_P0 [β_dn (P_n) > 0 for many data generating distributions P₀ and estimators of the optimal rule, though we have not analytically justified this claim. If the conditions of Theorem 5 hold, then this bias will only occur in finite samples. For simplicity we analyzed a different TMLE than the ones presented in this paper. First, we analyzed a TMLE for the single time point problem. We show in our online technical report that the single and multiple time point problems are closely related, so we expect that these results carry over to the two time point case. We have also analyzed a linear rather than logistic fluctuation in this section. We did this simply so we could get a straightforward expression for the bias of the TMLE without having to worry about linearizing the fluctuation submodel in a neighborhood of 0. Similar results should hold for the logistic fluctuations. We also assumed that E_P0 [Y|A = d_n(V), W] was estimated perfectly, which of course is not true in practice. Nonetheless, this assumption makes our results clearer because then we do not have to worry about a resulting empirical process term.

The term β_dn (P_n) only causes problems because d_n is learned from the same data over which the estimators of E_P0 Y_dn are run. The cross-validated approaches that we have presented in this paper do not suffer from this conjectured bias because we can condition on the training sample and then treat d_n as known. For fixed d, E_P0 [β_d(P_n)] = 0 and thus β_d(P_n) will not cause problems.