Using pilot data to size a two-arm randomized trial to find a nearly optimal personalized treatment strategy

Abstract

A personalized treatment strategy formalizes evidence-based treatment selection by mapping patient information to a recommended treatment. Personalized treatment strategies can produce better patient outcomes while reducing cost and treatment burden. Thus, among clinical and intervention scientists, there is a growing interest in conducting randomized clinical trials when one of the primary aims is estimation of a personalized treatment strategy. However, at present, there are no appropriate sample size formulae to assist in the design of such a trial. Furthermore, because the sampling distribution of the estimated outcome under an estimated optimal treatment strategy can be highly sensitive to small perturbations in the underlying generative model, sample size calculations based on standard (uncorrected) asymptotic approximations or computer simulations may not be reliable. We offer a simple and robust method for powering a single stage, two-armed randomized clinical trial when the primary aim is estimating the optimal single stage personalized treatment strategy. The proposed method is based on inverting a plugin projection confidence interval and is thereby regular and robust to small perturbations of the underlying generative model. The proposed method requires elicitation of two clinically-meaningful parameters from clinical scientists and uses data from a small pilot study to estimate nuisance parameters which are not easily elicited. The method performs well in simulated experiments and is illustrated using data from a pilot study of time to conception and fertility awareness.

1. Introduction

A personalized treatment strategy formalizes clinical decision making as a function that maps individual patient information to a recommended treatment. By selecting the right treatment for the right patients, personalized treatment strategies can lead to better patient outcomes while reducing overmedication, cost, and patient burden [1]. There is a growing interest in conducting randomized clinical trials when one of the primary aims is the estimation of an optimal personalized treatment strategy [2]. However, at present, there are no appropriate sample size formulae to inform the design of such a trial as existing methods treat estimation of a personalized treatment strategy as a secondary aim [3, 4, 5]. Furthermore, the expected outcome of the optimal personalized treatment strategy is nonregular (in the sense described in [6]); consequently, sample size formulae cannot be based on conventional asymptotic approximations of the sampling distribution of the estimated expected value of the optimal personalized treatment strategy [7, 8]. We propose a method for sizing a trial when the primary aim is the estimation of an optimal personalized treatment strategy that is based on inverting a plugin projection confidence interval [9, 10]. Because this interval is regular and robust to small perturbations of the underlying generative model so too is our sample size procedure. The method relies on auxiliary data in the form of a small pilot study or observational data and two simple clinician-elicited quantities. Simulations suggest that a pilot study with as few as 20 subjects may be sufficient to obtain reliable sample size estimators.

There is a large literature on estimating optimal treatment regimes using randomized or observational studies (see [11] and references therein). We consider a single treatment decision wherein baseline patient information is used to inform treatment choice; baseline information can include measures of disease stage severity, medical history, genetic and genomic information, as well as demographics, patient preferences and cost. The single decision problem is a special case of the more general problem of estimating a sequence of treatment rules, one per stage of clinical information, that map accrued patient information to a recommended treatment. Many existing estimation algorithms include single- and multi-stage variants. Regression-based methods, e.g., Q-learning [12, 13, 14, 15, 16], are a common and convenient tool for estimating interpretable treatment strategies from observational or randomized study data. Furthermore, extensions of these methods exist to accommodate a number of practical issues including: missing data [17], censored data [18], multiple outcomes [19, 20], binary outcomes [11], and high-dimensional data [21, 22]. Thus, we use Q-learning as the basis for constructing our sample size procedure. However, the proposed methodology could be extended to deal with other estimation procedures including direct value maximization [23, 24, 25, 26, 27], and A-learning [28, 29, 10, 16].

This work is partially motivated by our involvement with a clinical trial of fertility awareness and time to pregnancy (NICHD, K23 HD0147901, PI Joseph B. Stanford) and the planning of a follow-up trial. The original trial enrolled n = 123 subjects and demonstrated a promising but statistically insignificant benefit of a personalized treatment strategy. Furthermore, existing theory suggests that treatment response varies across subjects according to number of past successful pregnancies, length of sexual relationship, and other factors (see Section 5); thus, there is also scientific justification for a personalized treatment strategy in this context. Clinical investigators are therefore interested in running a follow-up trial with more subjects to estimate a high-quality personalized treatment strategy. The proposed methodology offers a principled method for sizing such a follow-up trial.

2. Q-learning

Q-learning is a simple regression-based method for estimating an optimal personalized treatment strategy. This is the assumed estimation method in the sample size calculations given in Section 3. However, we do not assume that the posited regression model used in the Q-learning algorithm is correctly specified. Let capital letters like X and Y denote random variables and lowercase letters like x and y denote instances of these random variables. To estimate a personalized treatment strategy we will collect n independent identically distributed triples $\mathcal{D}(n)={\{(X_i,A_i,Y_i)\}}_{i=1}^n$, one for each subject, where: X ∈ ℝ^p denotes baseline, i.e., pre-treatment, subject information; A ∈ {−1, 1} denotes the treatment received; and Y ∈ ℝ denotes the observed outcome for which higher values correspond to better clinical outcomes. We assume the simplest setting in which A is independent of X and assigned randomly with P (A = 1) = P (A = −1) = 1/2. The more general setting, in which the distribution of A depends on X, can be handled at the expense of are more complex expression for the marginal mean outcome under a pre-specified policy, e.g., see [6, 24]; the extension of asymptotic arguments to this setting are straightforward (see Remark 2).

A personalized treatment strategy is a map π : ℝ^p → {−1, 1} from the space of patient characteristics into the space of available treatments. Under π, a patient with X = x is recommended treatment π(x). To define the optimal treatment strategy we use the framework of potential outcomes [30, 31]. Let Y ^*(a) denote the potential outcome under treatment a ∈ {−1, 1}. For any treatment strategy, π, define Y^*(π) =Σ_a∈{−1,1} Y^*(a)1_π(X)=a to be the potential outcome under π. Define V (π) ≜ 𝔼Y^*(π), so that V (π) denotes the expected outcome if all subjects in the population are assigned treatment under π. An optimal treatment strategy, say π^opt, satisfies V (π^opt) ≥ V (π) for all π. To connect the potential outcomes with the observed data we assume: (i) Y = Y^*(A) (consistency); (ii) there exists c > 0 so that c < P (A = 1|X) < 1 − c with probability one (positivity); and (iii) {Y^*(−1), Y^*(1)} ⫫ A |X (no unmeasured confounders). Note that (ii) and (iii) are automatically satisfied by the randomized design considered here. However, if the pilot data (described below) are observational, then the validity of these assumptions must be considered carefully. Define Q(x, a) ≜ 𝔼 {Y^*(a)|X = x, A = a}, then, under the foregoing assumptions, it can be shown that Q(x, a) = 𝔼(Y |X = x, A = a) and V (π) = 𝔼Q {X, π(X)}. Thus, it can be seen that π^opt(x) = arg max_a∈{−1,1} Q(x, a).

In practice, the Q-function, Q(x, a), is not known. In Q-learning, an estimator of π^opt is formed by estimating Q(x, a) from the observed data and then using a plug-in estimator. We will assume a linear working model of the form $Q(x,a;\theta )=x_1^{\top }\alpha +a{x}_2^{\top }\beta$, where θ = (α^⊤, β^⊤)^⊤ and x₁, x₂ are known features derived from x. Note that x₁ and x₂ may contain polynomial terms or other nonlinear basis expansions. Let ℙ_n denote the empirical measure, i.e., given i.i.d. random variables Z₁, …, Z_n and function f defined on the support of Z, ${\mathbb{P}}_{n}f(Z)\triangleq n^{-1}{\sum }_{i=1}^{n}f(Z_i)$. Define least squares estimator θ̂_n ≜ arg min_θ ℙ_n {Y − Q(X, A; θ)}². Subsequently, ${\widehat{\pi }}_n(x)=arg{max}_{a\in \{-1,1\}}Q(x,a;{\widehat{\theta }}_n)=\text{sign}(x_2^{\top }{\widehat{\beta }}_n)$, is the estimated optimal treatment strategy.

Define θ^* ≜ arg min_θ 𝔼 {Y − Q(X, A; θ)}² and subsequently ${\pi }^{\ast }(x)\triangleq arg{max}_{a}Q(x,a;{\theta }^{\ast })=\text{sign}(x_2^{\top }{\beta }^{\ast })$. Thus, θ^* is the population analog of θ̂_n. However, π* need not equal π^opt if the working model for the Q-function is not correctly specified (see, for example, [14, 16, 32]). In the context of a sample size calculation 𝔼Y (π^*) is more relevant than 𝔼Y (π^opt) because the (possibly misspecified) postulated model for Q-function will be used in the analysis of the collected data. Our sample size procedure is robust in that provides sample size estimates that are correctly powered when this postulated model is incorrect.

2.1. Estimation of V (π^*) and nonregularity

For any γ ∈ ℝ^dim(x₂) let ${\pi }_{\gamma }(x)\triangleq \text{sign}(x_2^{\top }\gamma )$, and write V (γ) to denote V (π_γ). Applying a change of measure it follows that V (γ) = 2𝔼Y 1_AX^⊤γ≥0 [26, 24, 25]. Thus, V (β*) is a nonsmooth (nondifferentiable) functional of the underlying generative distribution [33, 34]. Consequently, there exists neither a regular nor an asymptotically unbiased estimator of V (β^*) [35, 8]. To see how this impacts sample size calculation, define

$${\widehat{V}}_{\mathcal{D}(n)}(\gamma )\triangleq 2{\mathbb{P}}_n\left\{Y{1}_{A{X}^{\top }\gamma \ge 0}-\left(1_{A{X}^{\top }\gamma \ge 0}-\frac{1}{2}\right)(X_1^{\top }{\widehat{\alpha }}_n+\mid X_2^{\top }\gamma \mid )\right\},$$ (1)

then V̂_𝓓(n)(β^*) is unbiased for V (β^*) and V̂_𝓓(n)(β̂_n) is an augmented inverse probability weighted estimator of V (β^*) [6]. Note that the first term in (1), 2ℙ_nY1_{Y X^⊤γ>0}, is a consistent estimator of V (γ) and is nonparametric in the sense that it does not rely on a model for the Q-function. The second term in (1), $-2{\mathbb{P}}_n(1_{A{X}^{\top }\gamma \ge 0}-1/2)(X_1^{\top }{\widehat{\alpha }}_n+\mid X_2^{\top }\gamma \mid )$ has mean zero and thus does not affect the consistency of V̂_𝓓(n)(γ), instead this term is included to reduce variability of the estimator. See [24] for a detailed derivation of this estimator in a more general setting. A standard approach to sample size calculation would depend on the limiting distribution of ${\mathbb{G}}_n\triangleq \sqrt{n}\left\{{\widehat{V}}_{\mathcal{D}(n)}({\widehat{\beta }}_n)-V({\beta }^{\ast })\right\}$. However, as stated above, because V (β^*) is nonsmooth, 𝔾_n is nonregular and its sampling distribution is difficult to approximate in finite samples (see, for example, [36] and references therein). In fact, V̂_𝓓(n)(β̂_n) need not even be consistent for V (β^*) (see [33]). Thus, sample size calculations should not be based directly on asymptotic approximations to the sampling distribution of 𝔾_n.

3. Projection sample size estimator

Let V₀ denote an elicited expected outcome under standard care, and let δ > 0 be an elicited value so that δV₀ denotes a clinically meaningful improvement in the expected outcome. For simplicity we treat V₀ as fixed, however, this is not essential, an alternative would be to replace V₀ with the mean outcome under a fixed treatment regime π(x) ≡ a for all x. Suppose that we observe data ${\mathcal{D}}_p={\{(X_i,A_i,Y_i)\}}_{i=1}^{n_p}$ from a preliminary study; this preliminary study could be a standalone pilot study, an internal pilot, or an observational study. For simplicity, we assume that 𝓓_p is collected in a randomized pilot study of exactly the same form as the follow-up study we are trying to size, but with far fewer patients (see Remark 3 below). Our goal is to determine a sample size n^* = n^*(𝓓_p) so that:

• (P1)The power to reject the hypothesis V (β^*) = V₀ in favor of V (β*) > V₀ is at least 1 − ρ when V (β^*) ≥ (1 + δ)V₀;
• (P2)P{|V̂_𝓓(n^*)(β̂_n^*) − V (β^*)| ≤ ε}≥ 1 − ρ;

  where 𝓓(n^*) is a follow-up study of size n^*, and ρ, ε > 0 are user chosen values. Conceptually, it is possible to have different values of ρ in (P1) and (P2), say ρ₁ and ρ₂; however, as we will show, because our method relies on inverting a confidence interval for V (β^*), it would be necessary to choose the level of this confidence interval to be {1 − min(ρ₁, ρ₂)} × 100 to ensure that (P1) and (P2) hold simultaneously so there is no loss in generality by setting ρ = ρ₁ = ρ₂. The condition (P1) ensures that, with high probability, the follow-up study will produce significant evidence that the optimal personalized treatment strategy leads to better patient outcomes on average than standard care when indeed the optimal personalized treatment strategy yields clinically significantly better outcomes. Thus, (P1) regards the viability of a data-driven personalized treatment strategy. (P2) ensures that the estimated optimal treatment strategy using data from the follow-up study will have an empirical performance near that of the optimal strategy. Thus, (P2) regards the quality of the estimated optimal treatment strategy. An alternative to (P2) is to try to control P(|V (β̂_n) − V (β^*)| > ε), however, there is no guarantee that a confidence interval for V (β^*) actually contains V (β̂_n). Because V (β̂_n) is a data-dependent parameter, constructing a confidence interval for this quantity is not well-studied (see [33, 34]).

Our sample size procedure works by constructing an estimator of the diameter of a confidence region for V (β^*) as a function of the sample size and then solving for a sample size sufficiently large to ensure (P1) and (P2) hold with high probability. Constructing a sample size based on confidence region diameter is sometimes categorized as precision-based or accuracy-based sample size calculation [37, 38]. Suppose that based on n observations we can construct a (1 − ρ) × 100% confidence interval for V (β^*), say ζ_n,1−ρ. Let d(n) denote the diameter of ζ_n,1−ρ. If d(n^*) ≤ δV₀, then V (β^*) and V₀ cannot simultaneously belong to ζ_n^*,1−ρ provided V (β^*) ≥ (1 + δ)V₀. Because P {V (β^*) ∈ ζ_{n^*,1− ρ}} ≥ 1 − ρ, we can distinguish V (β^*) from V₀ with probability 1 − ρ when V (β*) ≥ (1 + δ)V₀ by rejecting the hypothesis V (β) = V₀ when V₀ falls below the lower endpoint of ζ_n^*,1−ρ. Thus, (P1) is satisfied. Furthermore, if d(n^*) ≤ ε and V̂_𝓓(n^*)(β̂_n^*) ∈ ζ_n^*,1−ρ, then P{|V̂_𝓓(n^*)(β̂_n^*) − V (β^*)| ≤ ε} ≥ 1 − ρ, and (P2) is satisfied. Figure (1) shows a schematic illustrating the foregoing relationships between a confidence interval for V (β^*) and the criteria (P1) and (P2). To estimate n^* we first construct a valid (asymptotic) confidence interval for V (β^*) and an estimator d̂_𝓓p (n) of the diameter function d(n). Our estimator of n^* is given by n̂^* ≜ inf{n : d̂_𝓓p(n) ≤ min(δV₀, ε)}.

Figure 1.

Schematic showing how a confidence interval for V (β^*) can be used to ensure (P1) and (P2) hold. If the diameter of the confidence interval is less than (1 + δ)V₀, then the interval cannot simultaneously contain V₀ and V (β^*). Furthermore, if the diameter of the confidence interval is less than ε and V̂_𝓓(n^*)(β̂_n^*) belongs to the interval, then when the interval covers V (β^*) it follows that |V̂_𝓓(n^*)(β̂_n^*) − V (β^*)| ≤ ε.

To construct a confidence interval for V (β^*) we treat β^* as a nuisance parameter and construct a projection interval [9, 10]. For any fixed γ ∈ ℝ^dim(x₂) define

$${\sigma }^2(\gamma )\triangleq 4\mathbb{E}{(Y{1}_{A{X}^{\top }\gamma \ge 0}-\mathbb{E}Y{1}_{A{X}^{\top }\gamma \ge 0})}^2-4\mathbb{E}\{Y{1}_{A{X}^{\top }\gamma \ge 0}(X_1^T{\alpha }^{\ast }+\mid X_2^{\top }\gamma \mid )\}+4\mathbb{E}{\{(1_{A{X}^{\top }\gamma \ge 0}-1/2)(X_1^{\top }{\alpha }^{\ast }+\mid X^{\top }\gamma \mid )\}}^2,$$

and subsequently define the plug-in estimator

$${\widehat{\sigma }}_{\mathcal{D}(n)}^2(\gamma )\triangleq 4{\mathbb{P}}_n{(Y{1}_{A{X}^{\top }\gamma \ge 0}-{\mathbb{P}}_{n}Y{1}_{A{X}^{\top }\gamma \ge 0})}^2-4{\mathbb{P}}_n\{Y{1}_{A{X}^{\top }\gamma \ge 0}(X_1^{\top }{\widehat{\alpha }}_n+\mid X_2^{\top }\gamma \mid )\}+4{\mathbb{P}}_n{\{(1_{A{X}^{\top }\gamma \ge 0}-1/2)(X_1^{\top }{\widehat{\alpha }}_n+\mid X^{\top }\gamma \mid )\}}^2.$$

For any fixed γ ∈ ℝ^dim(x₂), under standard regularity conditions, it follows from the central limit theorem that $\sqrt{n}({\widehat{V}}_{\mathcal{D}(n)}(\gamma )-V(\gamma ))$ converges to a normal random variable with mean zero and variance σ²(γ). Therefore, an asymptotic (1 − μ) × 100% confidence interval for V (γ) based on pilot data 𝓓_p is

$${\mathbb{I}}_{\gamma }({\mathcal{D}}_p,n_p)\triangleq \left({\widehat{V}}_{{\mathcal{D}}_p}(\gamma )-z_{1-\mu /2}{\widehat{\sigma }}_{{\mathcal{D}}_p}(\gamma )/{\sqrt{n}}_p,{\widehat{V}}_{{\mathcal{D}}_p}(\gamma )+z_{1-\mu /2}{\widehat{\sigma }}_{{\mathcal{D}}_p}(\gamma )/{\sqrt{n}}_p\right),$$ (2)

where z_1−μ/2 is the (1 − μ/2) × 100 percentile of a standard normal random variable. Let $B\triangleq {(X_1^{\top },A{X}_2^{\top })}^{\top }$, define Ω = (𝔼BB^⊤)⁻¹𝔼BB^⊤ {Y − Q(X, A)}² (𝔼BB^⊤)⁻¹ to be the asymptotic covariance of $\sqrt{n}({\widehat{\theta }}_n-{\theta }^{\ast })$, and define Σ to be the submatrix of Ω corresponding to the asymptotic covariance of $\sqrt{n}({\widehat{\beta }}_n-{\beta }^{\ast })$. Let Σ̂_𝓓p denote the plug-in estimator of Σ based on the pilot data 𝓓_p. Define $\mathbb{T}({\mathcal{D}}_p,n_p)\triangleq \left\{\gamma :n_p(\gamma -{\widehat{\beta }}_{{\mathcal{D}}_p}){\widehat{\mathrm{\sum }}}_{D_p}^{-1}(\gamma -{\widehat{\beta }}_{{\mathcal{D}}_p})\le {\chi }_{\mathit{dim}(x_2),1-\xi }^2\right\}$, where ${\chi }_{\mathit{dim}(x_2),1-\zeta }^2$ is the (1 − ξ) × 100 percentile of a χ²-distribution with dim(x₂) degrees of freedom. Thus, 𝕋(𝓓_p, n_p) is an asymptotic (1 − ξ) × 100% confidence interval for β^*. An asymptotic (1 − ξ − μ) × 100% projection confidence interval for V (β^*) is therefore given by ⋃_{γ∈ 𝕋(𝓓p,np)} 𝕀_γ (𝓓_p, n_p). This interval has diameter d(n_p) = sup_{γ∈ 𝕋(𝓓p,np}) 𝕀_γ (𝓓_p, n_p) − inf_{γ∈𝕋(𝓓p,np}) 𝕀_γ (𝓓_p, n_p). For an arbitrary n ≥ n_p an estimator of d(n) is given by d̂_𝓓p (n) ≜ sup_{γ ∈ 𝕋(𝓓p,n) 𝕀γ} (𝓓_p, n) − inf_{γ∈𝕋(𝓓p,n)} 𝕀_γ (𝓓_p, n), and subsequently n̂^* = inf{n : d̂_𝓓p(n) ≤ min(δV₀, ε)}.

Remark 1

Our sample size procedure is based on choosing a sufficient number of samples to ensure that the width of a confidence interval is sufficiently small. It may be helpful to note that the proposed procedure is analogous to the following familiar example. Suppose we desire a confidence interval for the mean of a random variable Z of width no more than ω. Let Z̄_n denote the sample mean of random sample Z₁,…, Z_n. The natural (1 − ρ) × 100% confidence interval is ${\overline{Z}}_n\pm z_{1-\rho /2}\sigma \sqrt{n}$ which has diameter $d(n)=2z_{1-\rho /2}\sigma /\sqrt{n}$. Using pilot data 𝓓_p we construct estimator σ̂_𝓓p of σ and subsequently the estimator ${\widehat{d}}_{{\mathcal{D}}_p}(n)=2z_{1-\rho /2}{\widehat{\sigma }}_{{\mathcal{D}}_p}/\sqrt{n}$ of d(n); solving for n̂^* = inf{n : d̂_𝓓p (n) ≤ ω} gives the familiar expression ${\widehat{n}}^{\ast }=4z_{1-\rho /2}^2{\widehat{\sigma }}_{{\mathcal{D}}_p}^2/{\omega }^2$.

Remark 2

If the preliminary data are observational then the proposed sample size estimator can be applied with slight modification. To distinguish between the generative model for the pilot data and the data to be collected in the follow-up trial being sized, write ${\mathcal{D}}_p={\{(X_i^p,A_i^p,Y_i^p)\}}_{i=1}^n$. We assume that for any y, x, a P (Y^p ≤ y|X^p = x, A^p = a) = P (Y ≤ y|X = x, A = a), and P (X^p ≤ x) = P (X ≤ x). Furthermore, we require that the pilot study satisfy the consistency, positivity, and no unmeasured confounders assumptions stated in Section 2. Under these assumptions, the only difference between the population in the pilot study and the population in the randomized trial is the treatment assignment mechanism; note, weaker assumptions are possible. Let p(a|x) denote the treatment assignment mechanism P (A^p = a|X^p = x). Then,

$$\begin{array}{l}{\sigma }^2(\gamma )=8\mathbb{E}p(A^p\mid X^p){(Y^p{1}_{A^p{X}^{p\top }\gamma \ge 0}-2\mathbb{E}p(A^p\mid X^p)Y^p{1}_{A^p{X}^{p\top }\gamma \ge 0})}^2\\ -8\mathbb{E}p(A^p\mid X^p)\{Y^p{1}_{A^p{X}^{p\top }\gamma \ge 0}(X_1^{p\top }{\alpha }^{\ast }+\mid X_2^{p\top }\gamma \mid )\}+8\mathbb{E}p(A^p\mid X^p){\{(1_{A^p{X}^{p\top }\gamma \ge 0}-1/2)(X_1^{p\top }{\alpha }^{\ast }+\mid X^{p\top }\gamma \mid )\}}^2,\end{array}$$

and Ω = {𝔼p(A^p|X^p)B^pB^p⊤}⁻¹ 𝔼p(A^p|X^p)B^pB^p⊤{Y^p − Q(X^p, A^p)}² {𝔼p(A^p|X^p)B^pB^p⊤}⁻¹, and (α^*⊤, β^*⊤)^⊤= {𝔼p(A^p|X^p)B^pB^p⊤}⁻¹ 𝔼p(A^p|X^p)B^pY^p. Thus, with knowledge or an estimator of p(a|x) it is possible to construct plugin estimators of the quantities needed in the sample size formula with observational data.

Remark 3

The proposed sample size calculation does not require that the posited Q-function is correctly specified. However, if one is willing to assume that the Q-function is correctly specified then: (i) V (β^*) is still non-regular; and (ii) the proposed sample size procedure can be applied with a slight modification. Assume that $Q(x,a)=x_1^{\top }{\alpha }^{\ast }+a{x}_2^{\top }{\beta }^{\ast }$ so that the posited linear model is correctly specified. We assume X₁ has been centered so that 𝔼X₁ = 0. Then, $V(\beta )=\mathbb{E}Q\{X,{\pi }_{\beta }(X)\}=\mathbb{E}\{sgn(X_2^{\top }\beta )X_2^{\top }{\beta }^{\ast }\}$ and $V({\beta }^{\ast })=\mathbb{E}\mid X_2^{\top }{\beta }^{\ast }\mid$. Define ${\stackrel{\sim }{V}}_{\mathcal{D}(n)}(\gamma )={\mathbb{P}}_n\mid X_2^{\top }\gamma \mid$ then $\sqrt{n}({\stackrel{\sim }{V}}_{\mathcal{D}(n)}(\gamma )-\mathbb{E}\mid X_2^{\top }\gamma \mid )$ is asymptotically normal with mean zero and variance ${\tau }^2(\gamma )=\mathbb{E}{\{\mid X_2^{\top }\gamma \mid -\mathbb{E}\mid X_2^{\top }\gamma \mid \}}^2$. Define ${\stackrel{\sim }{\tau }}_{\mathcal{D}(n)}^2(\gamma )\triangleq {\mathbb{P}}_n{\{\mid X_2^{\top }\gamma \mid -{\mathbb{P}}_n\mid X_2^{\top }\gamma \mid \}}^2$ and

$${\stackrel{\sim }{\mathbb{I}}}_{\gamma }({\mathcal{D}}_p,n_p)\triangleq \left({\stackrel{\sim }{V}}_{{\mathcal{D}}_p}(\gamma )-z_{1-\mu /2}{\stackrel{\sim }{\tau }}_{{\mathcal{D}}_p}(\gamma )/{\sqrt{n}}_p,{\stackrel{\sim }{V}}_{{\mathcal{D}}_p}(\gamma )+z_{1-\mu /2}{\stackrel{\sim }{\tau }}_{{\mathcal{D}}_p}(\gamma )/{\sqrt{n}}_p\right).$$

Letting 𝕋(𝓓_p, n_p) be defined as above, then ⋃_{γ∈𝕋(𝓓p, np)} 𝕀̃_γ (𝓓_p, n_p) is a valid (1 − ξ − μ) × 100% valid (asymptotic) confidence interval for V (β^*). This interval can be inverted to construct a sample size. A potential drawback of this approach is that it depends heavily on the correctness of the postulated analysis model.

4. Simulation experiments

In this section we examine the finite sample performance of the proposed sample size procedure using simulated experiments. We use data from the following generative model:

$$\begin{array}{ll}{U}_i~\text{Bernoulli}(\nu ),\hfill & X_i~N_p(0,I_p),\hfill \\ Z_i=(I-U_i{P}_{{\beta }^{\ast }})X_i,\hfill & A_i~\text{Uniform}\{-1,1\},\hfill \\ e_i~N(0,1),\hfill & Y_i=X_i^{\top }{\alpha }^{\ast }+A_i{Z}_i^{\top }{\beta }^{\ast }+e_i,\hfill \end{array}$$

where P_β^* denotes the projection matrix onto the linear span of β^*. Thus, under this class of generative models ν × 100% of the subjects will have the same expected outcome regardless of which treatment they receive, i.e., the treatment effect for these patients is zero. We use a working model of the form Q(x, a, θ) = x^⊤α + ax^⊤β so that the working model is only correctly specified when ν = 0. In our experiments we fix p = 5 and α^* ≡ 1. We include intercept terms in our working models so the dimension of θ is 2p + 2 = 12. Furthermore, for any fixed value of ν ∈ [0, 1] we choose β^* ≡ c where c is chosen to be the smallest positive constant so that V (β^*) equals a pre-specified value. In our experiments we consider ν = 0, 0.05, 0.10, 0.25, 0.50, 0.75 and V (β^*) = 2, 2.25. We fix δ = ε = V₀ = 1. We set ρ = 0.80 and, following recommended best practice in [9], obtain an 80% confidence interval for V (β^*) by setting ξ = 0.01 thereby constructing a 99% confidence interval for β, and setting μ = 0.19 thereby constructing an 81% confidence interval for V (γ) for each fixed γ in the confidence set for β^*. For illustration, we also implement the sample-size formula ${\stackrel{\sim }{n}}^{\ast }({\mathcal{D}}_p)=4z_{1-(\alpha +\eta )/2}^2{\widehat{\sigma }}_{{\mathcal{D}}_p}^2({\widehat{\beta }}_{n_p})/{(\delta V_0)}^2$ which is a plugin estimator derived from the asymptotic normality of $\sqrt{n_p}\left\{{\widehat{V}}_{{\mathcal{D}}_p}({\beta }^{\ast })-V({\beta }^{\ast })\right\}$ discussed in Section 3. However, as discussed previously, normal approximations are not valid in this context without strong (unverifiable) assumptions (e.g., [35, 8]) and even when these conditions hold the small sample performance of these approximations may be poor [36, 7, 39, 33, 34]. A confidence interval using the normal approximation with data 𝓓 is ${\widehat{V}}_{\mathcal{D}}({\widehat{\beta }}_n)\pm z_{1-(\alpha +\eta )/2}{\widehat{\sigma }}_n({\widehat{\beta }}_n)/\sqrt{n}$.

We repeat the following procedure 1000 times. First, we generate pilot data 𝓓_p of size n_p = 20. Second, we use the sample size procedure described in the preceding section to obtain n̂^*(𝓓_p); because n_p is small, least squares estimation of all 2p + 2 parameters in our working model is unstable, hence in these experiments we use ridge-regression to estimate θ = (α^⊤, β^⊤)^⊤ with the amount of regularization chosen by BIC. The confidence intervals 𝕋(𝓓_p, n_p) are adjusted accordingly (details given in the appendix). Finally, we generate a follow-up study, say 𝓓^* = 𝓓 {n̂^*(𝓓_p)} and subsequently the projection interval ⋃_{γ∈𝕋{𝓓^*, n̂^*(𝓓p)}} 𝕀_γ {𝓓^*, n̂^*(𝓓_p)}for V (β^*). To approximate the union over all γ ∈ 𝕋{𝓓^*, n̂^*(𝓓_p)} we randomly sample 100,000 points from 𝕋{𝓓^*, n̂^*(𝓓_p)} using rejection sampling with a $\text{Normal}({\widehat{\beta }}_{n_p},n_p^{-1}{\widehat{\mathrm{\sum }}}_{{\mathcal{D}}_p})$ proposal distribution. Table 1 shows the estimated coverage, mean, and standard deviation of the proposed sample size procedure, as well as that of the normal-approximation-based estimator. The estimated coverage of V (β^*) is not significantly different at the 0.01 level from the target 0.80 in all but one case for the proposed estimator. Thus, empirically objective (P2) was attained. The resultant interval covered V₀ significantly less than the 1 − 0.80 = 0.20 target and thus the power to detect a difference between the adaptive strategy and standard care was significantly higher than the 0.80 target. Thus, empirically objective (P1) was attained. As expected, generally, both the mean and standard deviation of n̂^*(𝓓_p) increase as the proportion of individuals with zero treatment effect (ν) increases. Not surprisingly, the normal based confidence interval generally predicted smaller sample sizes than the proposed projection based procedure. However, the normal approximation sample size covered V (β^*) significantly less than the 0.80 target in 7 of the 12 generative models; furthermore, the normal based interval also included V₀ more frequently indicating less power.

Table 1.

Estimated coverage, mean, and standard deviation for the proposed sample size procedure based on 1000 Monte Carlo replications. Target coverage of V (β^*) is 0.80; bolded numbers indicate observed coverage is not significantly different from the target coverage at the 0.05 level.

				Projection sample size				Normal sample size
V*	V0	δ	ν	Cov V (β*)	Cov V0	𝔼n̂*(𝓓p)	SD (n̂*(𝓓p))	Cov V (β*)	Cov V0	𝔼n̂*(𝓓p)	SD (n̂*(𝓓p))
2	1	1	0	0.77	0.08	156	151	0.77	0.14	111	62
2	1	1	0.05	0.77	0.06	144	132	0.77	0.13	107	62
2	1	1	0.1	0.80	0.07	144	127	0.76	0.14	100	60
2	1	1	0.25	0.80	0.08	149	123	0.76	0.16	98	65
2	1	1	0.5	0.79	0.09	191	204	0.72	0.19	85	65
2	1	1	0.75	0.79	0.09	191	204	0.70	0.23	101	104
2.25	1	1	0	0.74	0.02	145	138	0.78	0.073	120	68
2.25	1	1	0.05	0.77	0.02	147	143	0.78	0.07	112	65
2.25	1	1	0.1	0.77	0.03	142	118	0.77	0.067	114	71
2.25	1	1	0.25	0.78	0.04	166	157	0.75	0.09	112	75
2.25	1	1	0.5	0.79	0.03	228	216	0.74	0.11	105	84
2.25	1	1	0.75	0.77	0.06	347	390	0.69	0.17	129	135

4.1. Midstream sample size adjustment

A common strategy to increase precision when sample size estimates are based on estimated nuisance parameters is to use an interim sample size adjustment based on updated estimates of the nuisance parameters [40]. We illustrate a simple interim update for our sample size procedure. First, we generate pilot data 𝓓_p of size n_p. Second, we use our sample size procedure to estimate n̂^*(𝓓_p). Third, we generate data 𝓓₁ in a follow-up study with ${\widehat{n}}_1^{\ast }({\mathcal{D}}_p)\triangleq [{\widehat{n}}^{\ast }({\mathcal{D}}_p)/2]$ subjects. Fourth, treating 𝓓₁ as a new pilot study, we collect data, say 𝓓₂, on an additional ${\widehat{n}}_2^{\ast }({\mathcal{D}}_1)\triangleq max\{0,{\widehat{n}}^{\ast }({\mathcal{D}}_1)-{\widehat{n}}_1^{\ast }({\mathcal{D}}_p)\}$ subjects. Let 𝓓^* ≜ 𝓓₁ ∪ 𝓓₂. Finally, we construct a confidence interval for V (β^*) based on the ${\widehat{n}}^{\ast }({\mathcal{D}}_p,{\mathcal{D}}_1)={\widehat{n}}_1^{\ast }({\mathcal{D}}_p)+{\widehat{n}}_2^{\ast }({\mathcal{D}}_1)$ subjects in 𝓓^*. Because our approach uses a projection interval, uncertainty about parameter values translates into larger diameters and hence larger estimated sample sizes. Consequently, the midstream update typically results in smaller sample sizes.

Table 2 shows the estimated coverage, mean, and standard deviation of the proposed sample size procedure based with a midstream update; to make the results comparable the table is constructed from the same 1000 Monte Carlo pilot data sets of size n_p = 20 used to construct Table 1. The mean and standard deviation of the total samples size n̂^*(𝓓_p, 𝓓₁) are both generally smaller when using the midstream update yet the coverages are comparable. This suggests that midstream update may be a viable strategy for reducing the number of subjects required to estimate an optimal personalized treatment strategy.

Table 2.

Estimated coverage, mean, and standard deviation for the proposed sample size procedure with a midstream adjustment based on 1000 Monte Carlo replications. Target coverage of V (β^*) is 0.80; bolded numbers indicate observed coverage is not significantly different from the target coverage at the 0.05 level.

V*	V0	δ	ν	Coverage V (β*)	Coverage V0	$\mathbb{E}{\widehat{n}}_1^{\ast }({\mathcal{D}}_p)$	𝔼n̂*(𝓓p, 𝓓1)	SD (n̂*(𝓓p, 𝓓1))
2	1	1	0	0.80	0.06	76	109	72
2	1	1	0.05	0.76	0.06	72	110	78
2	1	1	0.1	0.79	0.07	72	112	92
2	1	1	0.25	0.80	0.06	75	115	83
2	1	1	0.5	0.79	0.05	91	155	116
2	1	1	0.75	0.76	0.08	91	235	201
2.25	1	1	0	0.77	0.02	73	108	101
2.25	1	1	0.05	0.74	0.03	74	109	79
2.25	1	1	0.1	0.77	0.02	71	108	95
2.25	1	1	0.25	0.78	0.02	83	131	136
2.25	1	1	0.5	0.78	0.02	114	169	138
2.25	1	1	0.75	0.73	0.03	174	246	197

5. Time to pregnancy in normal fertility

A motivation for this work is our involvement with a randomized controlled trial of the effect of fertility awareness on time to pregnancy in normal fertility introduced in Section 1. We use data from this trial as pilot data to size a follow-up study. In this trial subjects were randomized to receive either standard care or follow the Creighton Model Fertility Care^™(CrM) System. The CrM involves patient education and daily patient self-monitoring to predict periods of fertility. Thus, there is an interest in identifying subjects for whom the burden of CrM is expected to be worthwhile. A personalized treatment strategy in this setting is map from baseline patient characteristics to a recommendation of CrM or standard care.

For simplicity, in our analysis we consider patients with complete information (n = 109 of an enrolled 123 subjects). Eligibility criteria required participants to be of proven fertility (full term pregnancy in past 8 years), and planning to conceive in the near future. For additional study details, see [41]. The outcome of interest is the minimum of eight cycles and the number of cycles before conception occurred. To be consistent with our development of finding strategies which maximize an expected outcome, we use eight minus the outcome of interest as our response. The two possible treatments are standard care (coded −1) and CrM (coded 1). Treatment was randomly assigned using a block-randomized design with blocking by age (blocks 18–29 and 30–35); thus, the no unmeasured confounders assumption (iii) of Section 2 is automatically satisfied. Variables thought to be predictive of benefit from CrM include number of previous successful pregnancies (Prg), subject age (years, Age), length of sexual relationship (years, SexLen), and education level (indicator of post-secondary, Edu). Let X denote these predictors, A the binary treatment, and Y the outcome. We use a linear Q-function of the form Q(x, a) = x^⊤α + ax^⊤β. Residual diagnostics for this model suggested that SexLen and Age should be transformed to the log-scale and that Prg should transformed to 1/Prg. Standard regression diagnostics are provided in the appendix. Table 3 provides the estimated coefficients for the Q-function. There is a significant interaction between CrM and the reciprocal of Prg suggesting that the benefit of CrM is greatest for subjects with a single prior successful pregnancy. The estimated optimal decision rule is π̂_n(x) = sign(x^⊤β̂_n). This rule has an estimated value of 5.82 (which corresponds to an average of 2.18 months to conception); in contrast the estimated value of assigning standard care to all patients is 5.52 (2.48 months to conception) whereas the estimated value of assigning CrM to all patients is 4.92 (3.08 months to conception). Thus, while there is some indication that a personalized treatment strategy might be effective, the evidence is not significant. For example, an 80% confidence interval for the value of the optimal decision rule, V (β^*), is [5.50, 6.13], which has a diameter of 0.63. It is of interest to know how this diameter would change in a larger, follow-up study. Figure 2 shows n̂^* as a function of the target diameter for an 80% interval. To see how this figure might be used, suppose that, based on the pilot study, one postulated that V (β^*) = 5.90, and V₀ = 5.52, the mean outcome treating all subjects. Then δ = V (β^*)/V₀ − 1 = 0.0688 so that a diameter of δV₀ = 0.38 is required for 80% power to detect superiority of V (β^*) to V₀. According to Figure 2, the estimated required size for a follow-up study is n̂^* = 280. If instead it was postulated that V (β^*) = 6.12, the required diameter is 0.5 and the required size for a follow-up study is n̂^* = 165. In addition to informing the size of a follow-up study, the slope of the line in Figure 2 can be viewed as a rough estimate of the incremental gain in the precision of V̂_𝓓(n)(β̂_n) per subject.

Table 3.

Estimated regression coefficients for linear Q-function. The outcome is coded so that higher values correspond to shorter time to conception. The is a significant interaction between CrM and 1/Prg suggesting that the benefit of CrM is greatest for patients with only one successful prior pregnancy (which was the minimum required for study eligibility). Indeed, all subjects assigned CrM by the estimated optimal treatment strategy had one successful prior pregnancy.

	Estimate	Std. Error	t value	Pr(>|t|)
Intercept	22.10	6.11	3.61	0.00
CrM	−1.80	6.11	−0.294	0.770
log Age	−5.72	1.99	−2.88	0.005
1/Prg	−0.586	0.642	−0.906	0.367
Edu	0.316	0.194	1.63	0.106
log SexLen	0.844	0.550	1.535	0.128
CrM:log Age	−0.0840	1.99	−0.042	0.966
CrM:1/Prg	1.47	0.647	2.26	0.025
CrM:Edu	−0.0920	0.194	−0.474	0.636
CrM:log SexLen	0.699	0.550	1.27	0.206

Figure 2.

Estimated sample size as a function of 80% confidence interval diameter. Reducing the interval width by a factor of two would require nearly a four-fold increase in sample size over the initial pilot study.

6. Discussion

We proposed a procedure for estimating the required sample size for a two-arm randomized clinical trial when the primary aim is the estimation of a personalized treatment strategy. The proposed method is based on inverting a projection confidence interval for the expected outcome under the optimal treatment strategy. Empirically the proposed method appeared to make efficient use of limited pilot data and enjoyed increased precision when assisted by a midstream interim update. Our procedure relied on standard assumptions for causal inference for treatment strategies, i.e., consistency, positivity, and no unmeasured confounders. The question of what happens when there are unmeasured confounders in the pilot is an interesting question but one that is difficult to answer in generality. The reason is that even in simpler settings, e.g., estimating an optimal regime from observational data, the impact of unmeasured confounding depends on the nature and severity of the confounding. Thus, characterizing the impact of unmeasured confounding on the sample size calculation would require first cataloging the effects of unmeasured confounding on estimation of the nuisance parameters across the many possible ways in which confounding can occur. It would be interesting future work to investigate the effects of confounding on the sample size procedure.

We focused on Q-learning as our method of estimation, however, there is growing interest in direct value maximization methods [42, 23, 24, 25, 26, 27, 43]. Distribution theory for parameters estimated via direct value maximization methods is lacking at present; when this theory is developed the proposed sizing procedure should be directly applicable. Furthermore, direct value maximization procedures often rely on a lower dimensional parameter than Q-learning, thus it is possible that the proposed sample size procedure will be less conservative when used with such methods.

Another important extension of this work is to sequences of treatment rules, also known as dynamic treatment regimes [28, 10]. Sample size procedures for sequential multiple assignment randomized trials [4] with the primary aim of constructing an optimal sequence of treatment rules are of increasing interest. The extension to multiple time points is complicated by the fact that in this setting the parameters indexing the Q-functions are nonregular [7, 32]. We are currently pursuing this extension.

7. Appendix

7.1. Wald type confidence region for ridge regression

For fixed nonnegative ridge parameter λ, let ${\widehat{\theta }}_n^{\lambda }=arg{min}_{\theta }{\mathbb{P}}_n{(Y-Q(X,A;\theta ))}^2+\lambda {\theta }^{\top }\theta$. Then, ${\widehat{\theta }}_n^{\lambda }={({\mathbb{P}}_{n}B{B}^{\top }+\lambda I)}^{-1}{\mathbb{P}}_{n}B{B}^{\top }{\widehat{\theta }}_n$. Recall θ = (a^⊤, β^⊤)^⊤, let U denote the constant matrix so that Uθ = β, and define $W_n^{\lambda }\triangleq U{({\mathbb{P}}_{n}B{B}^{\top }+\lambda I)}^{-1}{\mathbb{P}}_{n}B{B}^{\top }$, so that ${\widehat{\beta }}_n^{\lambda }=W_n^{\lambda }{\widehat{\beta }}_n$. It can be seen that $\sqrt{n}({\widehat{\beta }}_n^{\lambda }-W_n^{\lambda }{\beta }^{\ast })$ converges in distribution to a normal random variable with mean zero and variance-covariance matrix

$${\mathrm{\sum }}^{\lambda }=U{(PB{B}^{\top }+\lambda I)}^{-1}PB{B}^{\top }(Y-B^{\top }{\theta }^{\ast }){(PB{B}^{\top }+\lambda I)}^{-1}{U}^{\top },$$

which has plug-in estimator

$${\widehat{\mathrm{\sum }}}^{\lambda }=U{({\mathbb{P}}_{n}B{B}^{\top }+\lambda I)}^{-1}{\mathbb{P}}_{n}B{B}^{\top }(Y-B^{\top }{\widehat{\theta }}^{\lambda }){({\mathbb{P}}_{n}B{B}^{\top }+\lambda I)}^{-1}{U}^{\top }.$$

A Wald type confidence region for β^* based on ${\widehat{\beta }}_n^{\lambda }$ is given by

$$\left\{\beta :n{\left({\widehat{\beta }}_n^{\lambda }-W_n^{\lambda }\beta \right)}^{\top }{\left({\widehat{\mathrm{\sum }}}^{\lambda }\right)}^{-}\left({\widehat{\beta }}_n^{\lambda }-W_n^{\lambda }\beta \right)\le {\chi }_{\mathit{dim}({\beta }^{\ast }),1-\xi }^2\right\},$$

where the u⁻ denotes the generalized inverse of u.

7.2. Regression diagnostics for Q-function in fertility study

Figure (3) displays standard regression diagnostic plots using data from the fertility study.

Figure 3.

Regression diagnostics for fitted Q-function used in fertility study. The discreteness of the response manifests itself as parallel lines in the plot residuals vs. fitted values. Otherwise, there do not appear to be major departures from standard linear modeling assumptions.