Augmented trial designs for evaluation of principal surrogates

Abstract

Observation of counterfactual intermediate responses, and evaluation of them as candidate surrogates, is complicated in a standard randomized trial as half of the responses are systematically missing by design. Although some augmentation procedures exist for obtaining counterfactual responses, they are specific to vaccine trials. We outline extensions to the existing augmentations and suggest augmentations of three trial designs outside the setting of vaccines. We outline the assumptions needed to identify the causal estimands of interest under each augmented design, under which standard likelihood-based methods can be used to evaluate intermediate responses as principal surrogates. Two of these designs, crossover and individual stepped-wedge, allow for the observation of clinical endpoints under both treatment and control for some subset of subjects and can therefore improve efficiency over standard parallel trial designs. The third, the treatment run-in design, allows for the observation of a baseline measure that may be as useful a surrogate as the true counterfactual intermediate response. As the evaluation methods rely on several assumptions, we also outline a remediation analysis, which can be used to help overcome assumption violations. We illustrate our suggested methods in an example from a drug-resistant tuberculosis treatment trial.

1. Introduction

Causal effects are often discussed in terms of counterfactual responses, a response that would have occurred under a particular set of conditions sometimes counter to the actual conditions that were observed. In most cases, these counterfactuals cannot be obtained directly, but clever trial designs and estimands can be used to eliminate the need for them or to obtain measurements that can be used in place of the missing counterfactuals. The simplest example of a clever design is a blinded randomized placebo controlled trial. When randomization is properly conducted, all differences in outcome between the arms can be attributed to treatment assignment. This allows estimation of the marginal causal clinical treatment effect without the need to obtain the counterfactual clinical outcomes for each individual. How a clinical treatment effect varies over the levels of an intermediate response is often also of interest. If one conditions on the observed level of the intermediate response alone, the observed difference in clinical outcome between the arms is no longer causal. Treatment and control subjects that obtain the same level of the intermediate response may actually be very different if the treatment has any effect on the intermediate response. However, if we group subjects by their counterfactual level of the intermediate response, the comparison between the arms of the trial is causal within the groups.

Frangakis and Rubin (2002) introduce the concept of principal stratification (see also Robins, 1986), where the causal effect of treatment on the outcome can be evaluated over levels of a post-randomization intermediate response by considering the counterfactual values of the intermediate response under each treatment arm. Principal stratification is essentially a subgroup analysis where the subgroups are defined by the counterfactual intermediate response levels. This raises the question of whether there is an association between the intermediate response under treatment and the treatment effect on the clinical outcome. If this association is strong, the intermediate response is likely a good predictor of the treatment effect on the clinical outcome and can possibly be used as a surrogate for further treatment refinement and to predict clinical outcomes in the same setting. The investigation of such intermediate response measurements requires some way to measure or estimate the level of the missing counterfactual intermediate measure for each subject.

Follmann (2006) outlined two vaccine trial design augmentations, one to estimate the missing values and one to obtain an approximate measurement of the counterfactual intermediate response under vaccination for a subgroup of the placebo subjects. These augmentations can be used in the general clinical trial setting, as was suggested in Gabriel and Gilbert (2014). However, they are limited in concept because they were designed for vaccine trials with interventions that irrevocably change host immunity and absorbing endpoints of infection. In this paper, we extend these ideas beyond the vaccine trial setting to general interventions and endpoints. In the vaccine and absorbing event settings, special methods for estimation will always be needed due to the systematic pattern of missing counterfactuals, as outlined in the literature (Follmann, 2006; Gilbert and Hudgens, 2008; Gilbert and others, 2008; Qin and others, 2007; Huang and Gilbert, 2011; Li and others, 2010; Wolfson and Gilbert, 2010; Huang and others, 2013; Conlon and others, 2014; Gabriel and Gilbert, 2014; Gabriel and others, 2015). However, in other settings, with proper augmentation and assumptions, specialized methods are not needed for principal surrogate (PS) evaluation.

2. Notation and nomenclature

Let be the treatment indicator, 0 for control and 1 for treatment/intervention for subject ; we will focus on two levels of treatment for simplicity. Let be the observed value of the intermediate response variable for subject . As the intermediate response can be affected by treatment, there are two naturally occurring potential values of , under treatment and under control; let be the realization of the random variable for . Since the outcome of interest, , can be affected by both the treatment and the level of the intermediate response, it has levels defined by both, . We only consider the naturally occurring levels of defined by treatment, therefore, . In addition, one can consider different time periods when subjects might be on treatment or control given the previous history of treatment . We denote the counterfactual intermediate response during time period given the current treatment and the previous history of treatment by and the clinical outcome by . For example, a subject's second period counterfactual clinical response, had they been randomized first to treatment and then control, would be denoted as . We use a large number of abbreviations and acronyms throughout the paper and therefore provide Table 1 as a quick reference for these in one place.

Table 1.

Abbreviations and acronyms

ACN	Average causal necessity; the state of no clinical treatment effect when there is no treatment effect on the intermediate response.
BIP	Baseline immunogenicity predictor or baseline imputation and estimation predictor; a baseline measurement that is correlated with the missing counterfactual intermediate response of interest.
BIV	Baseline irrelevant vaccination; the procedure of giving a vaccine unrelated to the study vaccine to all subjects at the beginning of the trial. The immune response to this irrelevant vaccine can be used as a baseline immunogenicity predictor that is likely conditionally independent of the clinical outcome given the counterfactual intermediate response of interest, i.e. immune response to the experimental vaccine.
BSM	Baseline surrogate measure; the procedure of obtaining the intermediate response measure of interest at baseline.
CCT	Close-out control treatment; the procedure of treating control subjects at the end of follow-up with the study treatment for a given amount of time and obtaining a post treatment response, which under certain assumptions, this can be used as a direct imputation for the missing counterfactual intermediate response of interest.
COD	Crossover trial design; a study design under which subjects switch from their initially randomized study arm to the other to obtain intermediate and clinical outcome measurements under both trial arms.
CPV	Close-out placebo vaccination; the procedure of vaccinating uninfected placebo subjects at the end of follow-up and obtaining their post vaccination immune response measure, which under particular assumptions, can be used to impute the missing counterfactual intermediate response of interest.
RI	Run-in trial design; a study design under which all subjects are given treatment for a set amount of time prior to randomization to determine subject compliance, drug tolerability or to obtain response measures under treatment.
SUTVA	Stable unit treatment value assumption; the assumption that the counterfactual measures of the independent units of a study, for example subjects, are independent of other units' treatment assignments. A counterexample of this would be herd immunity.
WEM	Wide effect modification; the state of large differences in clinical treatment effect between groups defined by levels of the counterfactual intermediate outcome.

2.1. Principal surrogates

Frangakis and Rubin (2002) define an intermediate response to be a PS if no treatment effect on the intermediate measure corresponds to no treatment effect on the clinical outcome. This definition was refined in Gilbert and others (2008) for the vaccine trial setting. Gilbert and others (2008), as well as (Huang and others, 2013; Wolfson and Gilbert, 2010; Gabriel and Gilbert, 2014), suggest additional criteria for an intermediate response to be a good surrogate. All criteria are based either on the joint or marginal risk estimands conditioning on the potential intermediate response either under both the treatment and control or under treatment alone, respectively. We define the joint risk estimands of interest as some mapping of the cumulative distribution function of conditional on both counterfactual intermediate measures, , to the real line:

(2.1)

The counterpart marginal risk estimands are similarly defined by:

(2.2)

The arbitrary mapping allows for a variety of conditional risk estimands. Frangakis and Rubin (2002) define a PS by the criterion for all . Subsequent work added a second criterion that a good PS should have a risk contrast over the arms of the trial that varies widely in ; this has been called the wide effect modification (WEM) criterion in several works. We define WEM as there existing some such that , where the larger the the better the surrogate. Here is a causal effect predictiveness surface, i.e. a known contrast function satisfying if and only if , such as treatment efficacy ' (Gilbert and Hudgens, 2008).

One must find some way to identify these risk estimands to proceed with any of the methods of PS evaluation. Identification of the risk estimand requires conditioning on counterfactual values that are not observable for at least some subset of the subjects in a standard randomized trial. Figure S1 of the supplementary material available at Biostatistics online illustrates the uncertainty in the relationship between and , as both can never be observed. Under some set of assumptions linking the observed and counterfactual data, we can identify ; our suggested set of assumptions are outlined below. However, we need a way to observe or estimate the missing values for the control subjects to identify . All subjects are missing at least one counterfactual needed to identify the joint risk estimands, . We refer to any trial where the counterfactual intermediate response values can be accurately imputed or obtained as augmented. Throughout the paper we focus on the marginal estimands for simplicity. However, any of the methods developed could be used in both arms of the trial to identify the joint risk estimands.

3. Augmented experimental designs from the literature

After making two standard assumptions: stable unit treatment value assumption (SUTVA) and ignorable treatment assignment, formally given below, we are able to identify the marginal risk under treatment , but and both joint risk estimands remain non-estimable due to the missing counterfactual intermediate response measures.

• SUTVA: stable unit treatment value assumption, implies that observations on the independent units in the trial should be unaffected by the treatment assignments of other units,

• Ignorable treatment assignment: the observed treatment assignment does not change the counterfactual clinical outcome. This will generally hold in all blinded randomized trials.

Follmann (2006) outlined two vaccine trial design augmentations for inferring for placebo recipients, baseline irrelevant vaccination (BIV) [later generalized in Gilbert and Hudgens (2008) to baseline immunogenicity predictor (BIP)] and close-out placebo vaccination (CPV). Follmann (2006) suggests inoculating subjects with a vaccine unrelated to the study vaccine and measuring immune response to that vaccine as BIV. Follmann (2006) does note that any baseline measurement that is highly correlated with the immune response of interest and easily measured is useful and this is the concept explored and extended in Gilbert and Hudgens (2008) under the name of BIP. As pointed out in Follmann (2006), the measurement resulting from the BIV procedure, W , is unlikely to be associated with the clinical outcome after conditioning on the intermediate response of interest, allowing the risk model to be fit without inclusion of the BIV; . The more general BIP may be conditionally associated with the clinical outcome and should be considered for inclusion in the risk model.

The BIP augmentation procedure applies directly to the general clinical trial setting. Any baseline variable or group of variables useful for estimating the missing values is a BIP; ideally a BIP would satisfy . An additional baseline augmentation suggested in Gabriel and Gilbert (2014) is baseline surrogate measure (BSM), which is a baseline measurement of the candidate PS; . The BSM may be a good predictor of , with no further assumptions. Moreover, the BSM may be useful as a direct imputation of under assumption BSM SI, Surrogate time invariance at the Individual level, stated as:

• BSM SI: individual time constancy of the intermediate response from baseline to time after randomization under control, almost surely.

Assumption BSM SI can be rejected by simply observing pairs of measurements in the control arm for which . Under the BSM procedure the change in from baseline can be considered as a candidate PS for which, when BSM SI holds, the joint and marginal risk estimands are equivalent, as . An example of the use of this augmentation is given in Gilbert and others (2014).

A second augmentation, introduced by Follmann (2006), is CPV. Under CPV, placebo recipients uninfected at the end of the follow-up period, , are vaccinated, and their immune response, , is measured at time post vaccination. This measure is then used as a direct imputation of the missing measurement. Direct imputation is valid under two assumptions given in Wolfson and Gilbert (2010); see also Follmann (2006):

• CPV SI: individual time constancy of the immune response at time for placebo recipients with , almost surely;

• CPV NE: no infections during the close-out period, ; where is the indicator of an observed event after crossover and before is measured.

We will call the extension of CPV to the general clinical trial setting close-out control treatment (CCT); under which control subjects are given treatment at the end of the follow-up period and their intermediate response under treatment is observed. Fewer assumptions may be needed and greater efficiency obtained when CCT can be applied to all subjects and not just those uninfected at the end of the trial. We can use the measurements to directly impute missing measurements under the assumption:

• CCT SI: individual time constancy of the intermediate response at time post treatment under any amount of pervious control almost surely.

Assumption CCT SI extends CPV SI to the general case. In particular settings, such as in our example, an extension to CPV NE (no events) will not be needed as the outcome of interest is not event driven. In those cases where an assumption similar to CPV NE is needed for CCT to be used, the minimal necessary assumption will vary based on the setting. Panel (a) of Figure 1 illustrates a trial with BSM, BIP, and CCT augmentations. Although CCT, BSM, and generalized BIP help extend the concept of augmented trials outside of the vaccine trial setting, other maneuvers can be used when crossover is possible. We next suggest several augmentations of existing designs for obtaining observable proxies of the missing counterfactual intermediate responses.

Fig. 1.

Augmented trial design diagrams: panel (a) depicts the augmentations that have been previously described in the literature. Panel (b) depicts a crossover design where both the intermediate and clinical response measures are obtained after crossover. Panel (c) depicts a crossover design where subjects are only crossed over long enough to obtain their counterfactual intermediate response measures. Panel (d) depicts an active run-in trial design, where all subjects are given active treatment prior to randomization and their (intermediate) response is measured.

4. Extended augmented designs

4.1. Augmented crossover designs

Crossover trial designs (COD) have been used in select settings to help increase efficiency in estimating the treatment effect on the clinical outcome. Under an augmented crossover design, Panel (b) of Figure 1, the counterfactual measurements of the intermediate responses and a clinical outcome under either of the trial arms is observable for each subject. This design requires the assumption of no carry-over of treatment on the intermediate response measurements and the clinical outcome measurements, which may require a “washout” or pause before the 2nd period. We formalize the concept of “no carry-over” with the next two assumptions.

• COD SI: individual time constancy of the intermediate response under any treatment , over time regardless of the pattern of previous treatment: almost surely.

Assumption COD SI allows us to use the intermediate response information gathered after crossover for both groups. If we were merely crossing subjects over to observe their missing S(1) and S(0) levels, the full crossover follow-up would not be needed. The real advantage of the crossover design is in the measurement of the clinical outcomes, which can be used in the estimation of the conditional risk, Equations (2.1) and (2.2) under assumptions COD YD1 (Outcome Y time invariance in Distribution)

• COD YD1: distributional time constancy of clinical outcome within all subgroups defined by the intermediate response regardless of treatment history: for all and and .

When COD YD1 does not hold, the crossover design can be formulated as a double CCT design, Panel (c) of Figure 1. Both COD SI and YD1 may be implausible in some settings, and as the imperfect crossover measurements of and will never be measured on the same set of subjects as the true counterfactual responses, testing, and correction for violations can be difficult. We introduce two other designs, individual stepped-wedge and the treatment run-in design in the next sections. Both designs relax the COD SI and YD1 assumptions and allow for a greater amount of assumption testing and violation correction.

4.2. Augmented individual stepped-wedge

In settings where COD YD1 is only plausible for , a different assumption can be considered:

• COD YD2: distributional time constancy of clinical outcome under treatment after any length of control within all subgroups defined by the intermediate response, for all and .

When assumption COD YD2 holds, the distribution of clinical outcomes for those subjects with the same level that have been on treatment for a given amount of time during any period will be the same as it would have been if they were randomized to treatment in the first period. Under this assumption, as well as CCT SI and BSM SI, an individual stepped-wedge design can provide information about the counterfactual intermediate responses for all subjects and increase efficiency in estimation of the clinical treatment effect relative to a parallel trial. Under an individual stepped-wedge design, subjects are randomly selected to be in one of possible groups, where groups are defined by when subjects are crossed over to active treatment; Figure 2. This differs from the crossover design, as no subjects are returned to control after receiving treatment.

Fig. 2.

Individual stepped-wedge trial design diagram: figure depicts a two period individual stepped-wedge design with three arms where one-third of the subjects are randomized to be always on control, always on treatment or to switch from control to treatment at the end of the first period. One could also formulate this design as a multiple arm CCT, where the second group would only be crossed over long enough to have measured and no cross over outcomes, , would be observed.

Under assumptions CCT SI, COD YD2, and BSM SI, standard estimation methods can be used to estimate the conditional risk esimands, Equations (2.1) and (2.2), given that are measured for all subjects. When distributional time-constancy of the clinical outcome under control (COD YD2) is implausible, the individual stepped-wedge design can be formulated as a multi-arm multi-time point CCT design where randomly selected subjects are crossed-over at set time points and the intermediate response under treatment is measured, but no clinical outcome information is collected after crossover. This allows for greater testing of CCT SI, although these tests will still be distributional, in comparison to a standard single arm CCT design.

4.3. Augmented treatment run-in design

In settings where the assumption of individual time-constancy of the intermediate response (COD SI and BSM SI) is plausible, but time-constancy of the clinical outcome (under any treatment history, COD YD1, or under control alone, COD YD2) is implausible, a different assumption may be helpful:

• COD YD3: distributional time-constancy of clinical outcome after one period of active treatment within all subgroups defined by the intermediate response, for all and and

COD YD3 ensures that previous treatment for one period does not affect the distribution of clinical outcomes for subjects with the same levels of in the next period.

Under the COD SI and YD3 assumptions, a treatment run-in trial can be used to obtain the missing measurements in the control arm. A run-in (RI) on active treatment prior to randomization has been used in the past to identify individuals that are likely to be compliant or able to tolerate the treatment prior to selecting a randomization cohort. This design may also be useful for obtaining counterfactual intermediate responses. Panel (d) of Figure 1 depicts the trial design: all subjects receive treatment prior to randomization for time, at which point the intermediate response is measured. We will call this measurement . After a potential wash-out period, subjects are randomized to treatment or control and followed for the clinical endpoint. Under the treatment run-in design we have good power to test for individual time-constancy of the intermediate response, and . As outlined in the supplementary material available at Biostatistics online, with additional assumptions, this design could be used even in absorbing event settings.

5. Remediation analysis

Regardless of the evidence of assumption violations, we suggest that remediation analyses be undertaken whenever evidence of WEM is found. We focus on remediation for violations of individual time-constancy of intermediate responses; violations of clinical outcome time-constancy have less of an impact on bias and control of the Type I error rate, as demonstrated in the simulations.

Different designs use different proxies for the counterfactual in the control arm. These proxies include for the run-in design, for the crossover and stepped-wedge design, and for a 3-period step-wedge. For simplicity we will denote these measures by . Under the various individual time-constancy assumptions (i.e. BSM SI, CPV SI, CCT SI, COD SI), and we can fit the risk models directly. For the remediation analysis, we use an estimate of to estimate the conditional risk , under the conditional independence assumption for .

Fitting the model for those subjects that receive active treatment, , is straightforward. The likelihood contribution for those control participants with missing is obtained by integrating over the missing ,

(5.1)

Maximum likelihood estimation of the parameters of the conditional risk model can provide consistent estimates of the risk model parameters, provided the estimate is consistent. For example, one could assume a bivariate normal distribution for (or some transformation of them) with means , variances , and correlation and estimate these parameters in the run-in design for those subjects with observed values to estimate . Under the ignorable treatment assignment, this model will hold in the control arm and we can apply the estimated maximum likelihood (EML) procedure. For EML, is used in the likelihood to obtain estimates of the risk model parameters without maximizing over all parameters jointly (see Pepe and Fleming, 1991; Follmann, 2006; Gilbert and Hudgens, 2008). This could also be formulated as a measurement error model; more details are given in Section A of the supplementary material available at Biostatistics online. Although we focus on the marginal risk estimands here, Equation (2.2), the same procedure can be used in both arms for estimation of the joint risk estimands, Equation (2.1).

In practice, the distribution of can be estimated directly under the treatment run-in design, and with minor additional assumptions under the stepped-wedge design, depending on the assumed model. Estimation under the crossover design requires additional repeated measures to be taken and stronger assumptions made, as it is not possible to observe both on the same subject. Under any of the designs, the conditional independence assumption should be plausible, as the measurement is the same measure at a less relevant time point; under the treatment run-in design, this assumption can also be tested.

We suggest this procedure only when evidence of WEM is found under the analysis using the proxy as a direct imputation. Although this analysis controls the type 1 error rate well in settings where there is assumption violation, in settings where there is no violation it can induce type 1 error rate inflation. This can be a problem in the crossover setting, where it may seem desirable to assume that the association between and is conservatively low. See Table S4 of the supplementary material available at Biostatistics online.

6. Simulations

We illustrate our suggested designs for several scenarios by simulation. For ease of illustration we limit our examples to the marginal estimands, equation (2.2). We generate the outcome such that ; where is the true counterfactual measurement of the intermediate response under treatment during period 1 and and are both . The vector is taken to be or in the power and type 1 error rate settings, respectively. Repeated clinical outcomes are conditionally correlated 0.45 on average. The intermediate response was generated N(2,1). We assume trials with 100 subjects per arm with 2000 simulated trials for each scenario.

WEM requires , so we focus on this coefficient. We consider six scenarios for each of the three designs, the crossover, the individual stepped-wedge, and the treatment run-in, illustrated in Figures 1 and 2. We consider designs where subjects are crossed over only long enough to measure their missing counterfactual intermediate response for both the crossover and stepped-wedge designs. These designs are denoted as the “1st Period Only” in Tables 2 and 3. For the treatment run-in design, there are two ways to consider the intermediate response under treatment: use , the run-in measurement, for all subjects, denoted as “Run-in” in Tables 2 and 3, or use the observed response for those randomized to treatment after the run-in and for those randomized to placebo, denoted as “ imp” in the tables.

Table 2.

Bias and Type 1 error rate for under assumptions and using remediation analysis, bias (Type1 error rate %)

		Treatment run-in		Individual step-wedge		Crossover
Assumption violation	Model	imp	Run-in	1st only	All	1st only	All
	No Violations
None		0.003 (5.2%)	0.003 (5.2%)	(5.4%)	0.003 (5.2%)	(5.3%)	(5.4%)
	Remediation	0.004 (4.8%)	0.003 (4.8%)	(4.8%)	0.003 (5.2%)	0.081 (8.8%)	(5.6%)
	Violation of individual time constancy of intermediate response
White noise		0.249 (29.4%)	(4.3%)	0.293 (34.5%)	0.164 (31.3%)	0.258 (31.8%)	(5.2%)
	Remediation	0.062 (5.2%)	(4.5%)	0.074 (5.5%)	0.054 (5.7%)	0.062 (11.1%)	(5%)
Location shift		0.003 (5.2%)	0.003 (5.2%)	0.255 (—)	0.003 (5.2%)	(5.3%)	(5.4%)
	Remediation	0.004 (4.8%)	0.003 (4.8%)	(4.8%)	0.003 (5.2%)	0.081 (8.8%)	(5.6%)
	Violation of clinical outcome time-constancy
White noise		0.004 (4.8%)	0.004 (4.8%)	(5.4%)	0.004 (5.6%)	(5.3%)	(5%)
	Remediation	0.004 (4.9%)	0.004 (4.9%)	(4.8%)	0.005 (5.6%)	0.081 (8.8%)	(4.6%)
Risk shift		0.004 (4.8%)	0.004 (4.8%)	(5.4%)	0.004 (5.6%)	(5.3%)	(5%)
	Remediation	0.004 (4.9%)	0.004 (4.9%)	(4.8%)	0.005 (5.6%)	0.081 (8.8%)	(4.6%)
	Violation of both time-constancy assumptions
All		0.247 (21%)	(4.8%)	0.35 (46.2%)	0.166 (23.8%)	0.258 (31.8%)	(5.4%)
	Remediation	0.089 (5.6%)	(5.1%)	0.074 (5.5%)	0.07 (5.6%)	0.062 (11.1%)	(6%)

Each set of two columns are the bias and power for under a given design for the described set-up; each set of two rows are the analysis assuming the assumptions hold and then below it using the described remediation analysis. “1st only” uses the 1st period clinical outcome measurements and “All” using all outcome measurements, both of these estimation sets use all intermediate response measurements under treatment. In the treatment run-in design “ imp” refers to an estimation set where is imputed by for the placebo arm alone, and “Run-in” uses for all subjects. Bolded results indicate settings where type 1 error rate control was not maintained.

Table 3.

Bias and Power for under assumptions and using remediation analysis, Bias (Power)

		Treatment run-in		Individual step-wedge		Crossover
Assumption violation	Model	imp	Run-in	1st only	All	1st only	All
	No violations
None		0.003 (69.5%)	0.003 (69.5%)	(64.2%)	0.003 (94.2%)	(69.8%)	(99.6%)
	Remediation	0.004 (68%)	0.003 (68.4%)	(64.4%)	0.003 (94.2%)	0.005 (—)	(99.6%)
	Violation of individual time constancy of intermediate response
White noise		0.249 (—)	0.249 (37.8%)	0.293 (—)	0.039 (—)	0.256 (—)	0.123 (99%)
	Remediation	0.062 (35.6%)	(41.8%)	0.074 (29.8%)	0.006 (55.6%)	0.073 (—)	(91.5%)
Location shift		0.003 (69.5%)	0.003 (69.5%)	(96.6%)	0.003 (94.2%)	(70.8%)	(99.8%)
	Remediation	0.004 (68%)	0.003 (68.4%)	(64.4%)	0.003 (94.2%)	0.006 (—)	(99.8%)
	Violation of clinical outcome time-constancy
White noise		0.004 (51%)	0.004 (51%)	(64.2%)	0.004 (86.8%)	(69.6%)	(97.8%)
	Remediation	0.004 (50.9%)	0.003 (51.1%)	(64.4%)	0.004 (86.6%)	0.003 (—)	(97.6%)
Risk shift		0.004 (51%)	0.004 (51%)	(64.2%)	0.004 (86.8%)	(69.4%)	0.001 (97.9%)
	Remediation	0.004 (50.9%)	0.003 (51.1%)	(64.4%)	0.004 (86.6%)	0.011 (—)	(97%)
	Violation of both time-constancy assumptions
All		0.247 (—)	0.248 (27.6%)	0.35 (—)	0.04 (—)	0.247 (—)	0.129 (92%)
	Remediation	0.089 (21%)	(29.5%)	0.074 (29.8%)	(39.4%)	0.087 (—)	(75.1%)

Each set of two columns are the bias and power for under a given design for the described set-up; each set of two rows are the analysis assuming the assumptions hold and then below it using the described remediation analysis. “1st only” uses the 1st period clinical outcome measurements and “all” using all outcome measurements, both of these estimation sets use all intermediate response measurements under treatment. In the treatment run-in design “ imp” refers to an estimation set where is imputed by for the placebo arm alone, and “Run-in” uses for all subjects. Bolded results indicate settings where type 1 error rate control was not maintained.

Regardless of the outcome of the primary analysis, we apply a remediation analysis to each simulated trial assuming bivariate normal and proxy, and estimate the needed parameters to apply the EML procedure. This is always possible in the Run-in design and is possible in these simulations for the stepped-wedge design because we generate the data such that . It is not possible under the crossover design, and likely in practice under the stepped-wedge design, without additional repeated measurements on some subset of subjects. We therefore estimate the variance of in the crossover design by the absolute difference in the estimated variance of the observed and in Tables 2 and 3. Table 2 of the supplementary material available at Biostatistics online provides these same analyses based on a fixed conservative estimate under the crossover design.

The six scenarios we consider include: the perfect scenario where all assumptions hold, two scenarios where the observed proxy for the intermediate response changes over time by measurement error and by deterministic shift, two scenarios where clinical outcome changes over time by measurement error and deterministic shift, and a scenario where all four changes occur. The simulation scenarios are outlined in greater detail in the supplementary material available at Biostatistics online. Table 2 presents type 1 error rate and bias for and Table 3 presents bias and power for . As can be seen in the first row from both tables, when all assumptions hold perfectly, all the designs allow for unbiased estimation of the interaction term. As well, in this ideal setting the use of all outcome information increases power and reduces bias. In the first period only setting under the individual stepped-wedge design, approximately 66% of the subjects received control in the 1st period; power is expected to be reduced in comparison to the two other designs.

The increase in bias with the introduction of white noise to the proxy measurement of the intermediate response is as expected. These scenarios also have type 1 error rate inflation as can be seen in Table 2. One might expect that white noise error would cause bias toward the null; however, in this setting only those subjects with an imputed measurement have their measured with error. This biases the estimated association of and toward the null, while not affecting the estimated association in the treated group, biasing the interaction term away from the null. In the crossover setting where both outcome measures are used, the type 1 error rate control is maintained because both effect estimates suffer from the same bias toward the null, as all subjects contribute information to all estimated associations. Similarly, under the treatment run-in design when the run-in is used as a proxy for all subjects, type 1 error rate control is maintained.

The location shift in , e.g. for some constant , does not cause bias or power reduction in the crossover design or the treatment run-in design compared with the ideal setting. Although bias is increased when the second period outcomes are used under the crossover design, type 1 error rate remains nominal. Under the individual stepped-wedge design, the large bias caused by the shift violation in time-constancy of the intermediate response does cause an inflation of the type 1 error rate. Further investigation of this phenomenon suggests that it is not the amount of the shift that causes the type 1 error rate inflation, but that it occurs at more than one period in the trial; details are again outlined in Section A of the supplementary material available at Biostatistics online.

Violations of clinical outcome time-constancy will not affect the “1st Period Only” results, as by construction there is no assumption violation. For estimates using the second period clinical outcome measures generated under violations of clinical outcome time-constancy, both power and bias are adversely affected; however, not as dramatically as under violations of the time-constancy of the intermediate response. Type 1 error rate control is maintained under both types of violations of time constancy of the clinical outcome under all of the designs.

In practice, testing of the assumptions in prudent. Table S5 of the supplementary material available at Biostatistics online displays the power to detect violations of the individual and clinical time-constancy assumptions. The power is not high in many of the scenarios, particularly for the white noise violations. However, as can be seen in Tables 2 and 3, the remediation analysis helps control the type 1 error rate as well as reduces much of the bias in most of the scenarios where the assumptions are violated. This analysis requires good estimation of the error variance, which can be seen in the type 1 error rate inflation in all settings under the crossover design. However, using a conservative fixed estimate of provides proper type 1 error rate control when used only if the primary analysis supports WEM. As can be seen in Table S4 of the supplementary material available at Biostatistics online, this procedure can introduce asymmetric error in the properly measured . As well, in settings where poorly estimated, we generally see lower power and higher bias.

7. Example

A recent trial randomized 39 patients with extensively drug resistant tuberculosis (XDR-TB) to either immediate therapy with linezolid or a delay of 2 months, an example of a CCT design or a first period only stepped-wedge design with 1 step (Lee and others, 2012; Song and others, 2015). Shortly after drug introduction, the amount of circulating linezolid reaches steady state with a predictable trajectory between doses of drug. Our interest is in evaluating trough levels of linezolid as a PS for the outcomes of time to culture conversion (CC, essentially cure) and time to adverse events (AE).

Trough levels of linezolid under treatment are our intermediate response of interest, . Trough levels of linezolid are observed in the 1st period in the immediate arm, , but observed in the second period in the delayed arm, . We initially make the assumption that and fit a parametric Weibull regression with hazard

where , with indicating immediate treatment; the parameter is the shape. We find immediate treatment significant for culture conversion (; p-value 0.004), and trough value significant for AEs (; p-value 0.048). However, we do not find evidence of effect modification; , for either the CC or AE endpoints with p-values 0.20 and 0.46, respectively. As we find no evidence to support WEM in either model, we find no evidence to support trough levels of linezolid as a PS for either time to CC or AE. The above analysis relies on the individual time constancy of the intermediate response; . A Wilcoxon rank-sum test comparing the distributions of and has a p-value of 0.35.

We proceed with an illustrative remediation analysis, though in practice we only recommend doing so when WEM is supported. We assume bivariate normality of and directly estimate the mean and variance; bivariate normality is supported by a Q–Q plot. Less parametric methods can be used when normality is violated, such as those described in Huang and Gilbert (2011). We estimate by the correlation of the replicates comprising the averages and . We then implement EML as described in Section 5. The estimated parameters are () for CC and () for AE, making it no longer significant, while remain non-significantly different from zero in both models. Standard errors are based on a variance estimator applied to the nonparametric bootstrap samples. As expected, the for CC remains very similar to the original estimate, while the changes more substantially. The SE for both increase as expected and the variance ratio for from standard to the remediation model is 1.46 and 2.44 for CC and AE, respectively. We also applied EML using a conservative and found very similar results.

8. Discussion

We outline three types of augmented design extensions: the crossover design, the individual stepped-wedge design, and the treatment run-in design, that can be used in different settings where different sets of assumptions are more likely to hold. Each design has potential benefits and drawbacks. These extensions introduce the idea that principal stratification and principal surrogate evaluation need not be confined to the vaccine trial setting and that outside that setting counterfactual measurements are potentially easier to obtain, as can be seen in the example. These designs provide advantages over standard randomized trials and previously suggested augmented designs for identifying and estimating the intermediate response-dependent risk estimands by allowing standard methods of analysis to be used and increasing power.

As is illustrated in the simulations, violations of the individual time-constancy assumptions can cause bias and type 1 error rate inflation under some of the designs. However, as we outline in Section 5 and illustrate in the simulations, most violations can be overcome using our suggested remediation analysis. For simplicity we outlined the remediation analysis using a fully parametric specification, analogous semi-parametric, and non-parametric formulations, as well as pseudo-score methods, can be used for the same purpose (Huang and Gilbert, 2011; Gilbert and Hudgens, 2008; Huang and others, 2013).

Although not a main focus of this paper, surrogate consistency has been discussed in many works and more recently has become a hot topic in the surrogate evaluation literature (VanderWeele, 2013; Elliott and others, 2015). A consistent surrogate will not wrongly predict the direction of the true clinical effect on the individual level, an important problem to consider when evaluating a surrogate as a target for treatment improvement (Chen and others, 2007). Although empirical assessments of consistency was introduced by Wu and others (2011), actually observing clinical outcomes under both treatment arms for an individual could only improve our understanding of the relationship between the surrogate and the clinical outcome.

Supplementary material

Supplementary Material is available at http://biostatistics.oxfordjournals.org.