Predictive cluster level surrogacy in the presence of interference

Summary

Surrogate evaluation is a difficult problem that is made more so by the presence of interference. Our proposed procedure can allow for relatively easy evaluation of surrogates for indirect or spill-over clinical effects at the cluster level. Our definition of surrogacy is based on the causal-association paradigm (Joffe and Greene, 2009. Related causal frameworks for surrogate outcomes. Biometrics65, 530–538), under which surrogates are evaluated by the strength of the association between a causal treatment effect on the clinical outcome and a causal treatment effect on the candidate surrogate. Hudgens and Halloran (2008, Toward causal inference with interference. Journal of the American Statistical Association103, 832–842) introduced estimators that can be used for many of the marginal causal estimands of interest in the presence of interference. We extend these to consider surrogates for not just direct effects, but indirect and total effects at the cluster level. We suggest existing estimators that can be used to evaluate biomarkers under our proposed definition of surrogacy. In our motivating setting of a transmission blocking malaria vaccine, there is expected to be no direct protection to those vaccinated and predictive surrogates are urgently needed. We use a set of simulated data examples based on the proposed Phase IIb/III trial design of transmission blocking malaria vaccine to demonstrate how our definition, proposed criteria and procedure can be used to identify biomarkers as predictive cluster level surrogates in the presence of interference on the clinical outcome.

1. Introduction

There are many definitions of surrogacy, but surrogacy is generally only considered for two levels, individual and trial. Individual level surrogates attempt to predict individual level outcomes and efficacy, although it is actually a prediction of the average outcome or efficacy for a subgroup having the same level of the biomarkers. Trial level surrogates are used to predict trial level outcomes. Surrogate evaluation generally assumes independence of the units at the level of evaluation, individuals or trials. The desired level of surrogacy is unclear when clinical efficacy can only be measured at the cluster or community level. In cases, where there is no direct individual clinical benefit, for example, when all benefit is only through herd immunity, the outcome at the community or cluster level is the only meaningful clinical outcome. However, as evaluation of surrogates for total or indirect effects may be of interest for any intervention with spill-over effects, our development is likely applicable in almost all infectious diseases settings.

In our motivating setting of malaria transmission blocking vaccines (TBV), vaccines aim to induce sexual stage antibodies in a human host that provide little to no direct benefit to the vaccinated subject. The malaria parasite life-cycle requires a sexual stage inside a mosquito. The hope is that antibodies induced by vaccination will be taken up with a blood meal and block parasite development in the mosquito, halting transmission to another human host. In this setting, cluster or area randomized trials are needed to estimate efficacy. Cluster randomized trials can be expensive and difficult to run, thus a high quality surrogate in this setting would be valuable to eliminate useless vaccines and provide targets to improve promising vaccine candidates prior to Phase III trials.

Most individual level surrogate evaluation methods implicitly or directly make the assumption of no interference between individuals, a major component of the stable unit treatment value assumption (SUTVA) (Rubin, 1980). At the subject level, SUTVA is clearly violated in our motivating setting. Even when SUTVA holds, individual level surrogacy can be difficult to evaluate in a principled manner due to confounding or the lack of identifiability of the causal estimands of interest. Unlike the individual level, trial level causal effects on the clinical and surrogate outcome can be observed simultaneously. At the trial level, vaccine efficacy, for example, can be observed in a randomized trial, as can the ratio of the proportions of subjects with a detectable immune response on each arm. Therefore, associations between these causal effects can be assessed over a set of trials from observable data.

In our setting of interest, the only meaningful measure of clinical outcome is at the cluster or group level; the desired comparison between two treatments within a given cluster cannot be observed. However, for certain candidate surrogate measures, and under assumptions we will outline below, it is possible to observe some useful causal contrasts of the surrogate at the cluster level. Although the concept of trial level surrogacy applies in our motivating setting, given the expense and logistical difficulty of a cluster randomized trial, it is unlikely an adequate number of such trials will exist to evaluate surrogates at the trial level. Furthermore, the trial level treatment effect on a surrogate averaged over all clusters in a cluster randomized trial is unlikely to be comparable to treatment effects on the surrogate observed in small early phase trials.

To provide a concrete example, consider our motivating setting in which we randomize 8 clusters to either 10% or 90% coverage with vaccine, at a ratio of one-to-one, and within each cluster randomize subjects to vaccine or placebo according to the assigned coverage proportion. In each cluster, we count the number of clinical malaria cases giving us the clinical outcome of interest. For each individual, we measure an immune response specific to vaccine prior to the start of the wet season, before any clinical malaria has occurred, but after the last vaccination is given. We calculate the difference in average immune response for subjects receiving vaccination and those not receiving vaccination within each cluster. This gives us a cluster level causal difference in the candidate surrogate. We can therefore directly estimate a causal contrast in the surrogate. Ideally, we would also be able to directly estimate a causal contrast in the clinical outcome and investigate the association between these two effects over the clusters. However, as this is not of interest in our motivating setting, or in any setting where the clinical effect of interest is the indirect or total effect, we propose a method of surrogate evaluation, which can be used in our settings where the primary clinical outcome is a spillover or indirect effect at the group or cluster level.

To the knowledge of the authors, the only similar methods in the presence of interference consider mediation (VanderWeele, 2010; Vanderweele and others, 2013). Although mediation analysis similar to those suggested in VanderWeele (2010), Vanderweele and others (2013), as well as in other literatures (Zhang and others, 2009), may be applied in our setting with only minor extension, it may not be conceptually applicable in our, and many settings like it, for two reasons. First, in order for mediation analysis to be sensible, one must conceive of a way to intervene on the candidate mediator independent of the intervention, providing a way to effectively randomize the mediator. We do not see how one might do this in our motivating setting, but also in general for many vaccine studies in naive subjects. Secondly, although finding a surrogate that is also in the causal pathway is ideal, we do not believe that many of our candidate surrogates will actually be in the causal pathway. Instead, we are concerned with good predictors of clinical effects or effect modifiers of the clinical treatment effect that provide targets for intervention improvement. For example, in our motivating setting, the standard membrane feeding assay, described in detail in the simulated example section, is attempting to measure something that is very likely in the causal pathway, but is clearly a simplified measure of a complex characteristic. This is true for many similar settings where a high quality predictive surrogate may be a poor mediator because it fails to isolate or fully measure the characteristic that is in the causal pathway.

We instead focus on what Joffe and Greene (2009) called the causal-association paradigm. Under this paradigm, regardless of the level of surrogacy, our main interest is in the association between some causal effect on the clinical outcome and some causal effect on the candidate surrogate outcome. Surrogate evaluation methods under this paradigm have been developed in the principal stratification line of literature at the individual level (Frangakis and Rubin, 2002; Follmann, 2006; Gilbert and Hudgens, 2008; Wolfson and Gilbert, 2010; Huang and Gilbert, 2011; Huang and others, 2013; Gabriel and Gilbert, 2014; Gabriel and others, 2015; Taylor and others, 2015), and in the meta-analysis literature at the trial level (Daniels and Hughes, 1997; Gail and others, 2000; Buyse and others, 2000; Burzykowski and others, 2001; Dai and Hughes, 2012; Gabriel and others, 2016).

In this article, after outlining the necessary notation and the estimands of interest in Section 2, we describe the concept of cluster level surrogacy in the presence of interference and give a definition of a predictive cluster level surrogate under the causal-association paradigm in Section 3. In this same section, we propose criteria for evaluating candidate surrogates for this definition. In Section 4, we state a set of assumptions that allow us to link the observable data to the desired estimands in our proposed trial setting, before outlining our suggested method of estimation. In Section 6, we base our simulations on the proposed Phase IIb/III trial design of Delrieu and others (2015). Finally, we discuss how the most common cluster randomized trial designs can still be used for surrogate evaluation in Section 7 before summarizing and considering future areas of research in Section 8.

2. Notation

Suppose there are clusters of individuals. For a cluster , let denote the number of subjects who are indexed by . Let the treatment for the individuals in cluster be denoted by the vector ; with the sub-vector excluding the treatment of subject written as . The received treatment for subject in cluster takes on one of two possible values 0, for placebo, or 1, for treatment. Let be the coverage proportion in cluster , which is the proportion of subjects receiving treatment in cluster . For simplicity, we consider two coverage levels of the same intervention, which we will denote as and and follow closely the notation of Hudgens and Halloran (2008). We also define the variable where if cluster is randomized to coverage level and if randomized to coverage level ; . Let the observed individual clinical outcome be denoted by and the observed intermediate outcome, or candidate surrogate, be denoted by .

Let be the potential clinical outcome of subject in cluster if, potentially counter to fact, the treatments were to be for all subjects in cluster other than subject and for subject . This is the hypothetical outcome had the randomization been and ; we will later make assumptions that allow one to link these potential outcomes to the observable data. Defining the potential outcomes in this way allows the th subject’s outcome to be influenced by both their treatment and the treatment of other subjects in the same cluster, . For this reason, even our notation implicitly makes the assumption that there is no interference between clusters. Let us denote the potential intermediate outcome for a subject in cluster if their personal treatment were to be and the rest of cluster had been given treatments by .

Although we allow for the potential outcomes to be influenced by any treatment of the other subjects within the cluster, we will focus on strategies that randomize treatment according to one of the coverage levels of interest, or . Let the set of all possible randomizations for a cluster of size that satisfy some coverage level be denoted as and the subset of these randomizations such that any given subject’s treatment is be denoted by . We can then define the individual average clinical outcome for subject had they received treatment within cluster under coverage level by

We will use as shorthand for . This is the conditional average of had subject received under coverage for cluster ; where is the probability of randomizing to treatment vector with , given all possible treatment assignments that follow coverage . The exact expression for this probability, , will depend on the type of randomization. One could also consider the outcome marginalizing over the individual randomization, , , to investigate overall effects, e.g. , but we will focus on the conditional outcomes for clarity.

We can then define the clinical estimands at the cluster level as: and Then the cluster average direct effects on the intermediate outcome under coverage level are:

(2.1)

We will consider the cluster level outcomes to be independently and identically distributed over the clusters. Thus, we can define the population average clinical outcome for subjects under treatment and coverage level by , where the expectation is taken with respect to the underlying data generating mechanism or super-population of the clusters.

Based on these population average estimands, we can define causal contrasts between the coverage levels. For example, we denote the population indirect effect on the clinical outcome by . This is the effect on the clinical outcome of coverage when compared with coverage for untreated subjects. The population total effect is the difference in the population average clinical outcome for treated subjects under coverage level compared with the population average clinical outcome for untreated subjects under coverage . We denote this as . A depiction of the coverage levels and the comparisons of interest between the clusters is given in Figure 1.

Fig. 1.

Coverage and causal effects diagram.

3. Cluster level surrogacy

We are interested in biomarkers that can serve as targets for intervention improvement and allow for better early phase decision making, as well as good predictors of treatment effects on the clinical outcome. Therefore, we are interested in the association between causal effects on the clinical outcome at the cluster level and causal effects on the surrogate outcome at the cluster level.

Predictive cluster level surrogate: If some cluster average effect of treatment on an intermediate outcome is predictive of some cluster average effect on the clinical outcome within a specific trial, then that intermediate outcome is a predictive cluster level surrogate for that cluster average effect for the specific setting of that trial.

Here, we emphasize the setting specific nature of the surrogate because it is only evaluated within the given trial setting, rather than over a group of different settings or in a new setting. This distinction is described in detail for individual-level surrogates in Gabriel and others (2016). As well, although this definition allows for any causal effect on the surrogate to be predictive of any causal cluster average effect on the clinical outcome, here we are especially interested in predicting cluster average indirect effects by the cluster average direct effect on the intermediate outcome. As we cannot observe the cluster average outcomes under both coverage levels, we propose to evaluate predictive cluster level surrogates in the same way that principal surrogates, at the individual level, are often evaluated.

For the evaluation of surrogacy, we estimate one of the population average clinical outcomes, outlined above in Section 2, conditional on the cluster level direct effect on the intermediate outcome. The cluster level clinical outcome under treatment and under coverage level conditional on the direct effect on the surrogate under both coverage levels is given by: A similar conditional estimand is suggested in Gilbert and Hudgens (2008) for the evaluation of principal surrogates. It is contrasts between these that are of interest for surrogate evaluation as described in Gilbert and Hudgens (2008), via the causal effect predictiveness (CEP) surface or curve. A CEP surface or curve is any function such that if and only if , where here we will consider and that are the population average outcomes conditional on the direct effect(s) on the candidate surrogate. Conditioning on both and makes this conditional estimand difficult to identify without strong assumptions, but also makes the resulting evaluation less useful as we discuss below. If we can assume no interference on the surrogate we can identify the conditional estimands with a few additional assumptions, but no additional measurements (Wolfson and Gilbert, 2010).

Under the assumption of no interference on the surrogate, we can construct a CEP curve based on the indirect effect on the clinical outcome conditional on the direct effect on the candidate surrogate. For example, a CEP for the indirect effect comparing coverage and , , can be defined as , which is a function that varies with the value of and is only equal to zero when . One can similarly define as a CEP curve, by . We will focus on without loss of generality.

Although no interference on the surrogate may seem like a strong assumption, in our motivating setting it is plausible as interference is unlikely for many of the candidate surrogates of interest. For example, antibody response 6 weeks post last vaccination and standard membrane feeding will not be influenced by the vaccination or outcome of other subjects.

Similar logic holds for any vaccine that does not cause active shedding of the antigen by the vaccinated. Antibodies in an individual induced by flu vaccine prior to flu season are unlikely to be influenced by the vaccination of surrounding individuals. Antibodies induced by an HIV vaccine will not be dependent on the vaccination of others, while HIV infections are very likely to depend on the vaccination of sexual partners. Whether the assumption of no interference on the surrogate is plausible depends on the surrogate measurement and the nature of the clinical outcome. In infectious disease, in particular, there are many examples of settings where there will be interference on the clinical outcome of interest, but not on the candidate surrogate.

Using the conditional estimands, we propose the use of similar criteria as those used for predictive individual-level surrogate, or principal surrogate, evaluation.

• Average causal necessity (ACN): there is no effect on the average clinical outcome of interest when there is no direct effect on the candidate surrogate, i.e. (Gilbert and Hudgens, 2008).

• One-sided strong average causal sufficiency (1SSACS): without loss of generality, let effects indicate benefit for the clinical outcome and effects indicate benefit on the candidate surrogate outcome. Then higher coverage implies benefit for all positive direct effects on the candidate surrogate, i.e. for the CEP for all . (Gilbert and Hudgens, 2008).

• Wide effect modification (WEM): the CEP curve varies in , i.e. there exists some such that (Gabriel and others, 2015).

The criterion WEM is a less restrictive and more easily verified alternative criterion than ACN and 1SSACS. Any variation in the CEP curve over is evidence of WEM, and provides a target value of a surrogate that can be used for intervention improvement and early phase decision making; criteria ACN and 1SSACS together are a special case of WEM. When ACN and 1SSACS can both be verified, this is a stronger validation of a surrogate. However, for a candidate surrogate to have any value as a predictive surrogate only WEM is needed, as this gives a target for intervention improvement. For the evaluation of candidate surrogates based on any of the criteria using observable data, we will need to make some additional assumptions.

4. Assumptions

To evaluate the criteria for a predictive cluster level specific surrogate, we first need to make assumptions to link the observable data to the desired counterfactuals and causal estimands. One set of such assumptions is as follows:

• Assumption (a): Consistency, as discussed in Cole and Frangakis (2009).

• Assumption (b): Cluster level outcomes are independently and identically distributed (IID).

• Assumption (c): because there is no interference on the surrogate.

• Assumption (d): All assignment strategies are mixed as defined in Hudgens and Halloran (2008).

• Assumption (e): Stratified interference as defined in Hudgens and Halloran (2008).

In assumption (a) “consistency” means that we assume what is observed given the realized randomization is the potential outcome under that intervention, i.e. at the individual level . The independence in Assumption (b) is plausible in our setting and can be tested for and encouraged with proper trial design. The cluster level outcome being an IID random sample is the same assumption made when using random effects for a clustered data problem. We highlight this as an assumption, to make clear the difference between the population estimators used here and those of Hudgens and Halloran (2008). However, we select cluster level estimators allowing for individual level potential outcomes within cluster to be fixed, similar to Hudgens and Halloran (2008). Assumption (c) is described in detail above and allows us to identify the estimands of interest. In practice, assumption (c) cannot be directly tested. However, a testable null hypothesis, which if rejected provides evidence against assumption (c), is that the marginal distributions and are the same.

By mixed in Assumption (d), we mean all clusters have the same probability of being randomized to a given coverage and exactly clusters will receive coverage and coverage . As well, within a cluster, each individual has the same probability of receiving treatment, given the coverage, and exactly subjects will get treatment in cluster such that is as close to as possible (Sobel, 2006; Hudgens and Halloran, 2008). Assumption (e), Stratified interference, is an assumption outlined in both Hudgens and Halloran (2008) and Tchetgen and VanderWeele (2012). It states that only a subject’s treatment and the coverage level within their cluster impacts their clinical outcomes. Therefore, all possible counterfactuals will have the same value for all . This means that the observed are equal to if the observed .

5. Estimation and inference

5.1. Within cluster estimation

As we make no assumptions about the distribution of the individual data within clusters, we propose to use the distribution free cluster level estimators of Hudgens and Halloran (2008):

(5.1)

and , These estimators were shown to be unbiased in Hudgens and Halloran (2008) under assumptions (a–d). In the finite population setting, assuming individual outcomes are fixed conditional on cluster, an additional assumption is needed to use the variance estimator of Hudgens and Halloran (2008) for either estimate. Under Assumptions (a–e), one can derive an unbiased estimator for the variance of the estimator given above and a conservative variance estimate for estimator 5.1 (for illustration of why this is conservative for fixed potential outcomes, or in the finite population of the selected cluster , see Theorem 5 of Hudgens and Halloran (2008)).

The same and estimators could be used if one is willing to assume that the individual level outcomes are random and drawn from some possible cluster-specific distribution, which would allow for variance estimation. Under this scenario, one could also relax assumption (e). In particular, if one is willing to assume some model for the interference within cluster, a parametric or semi-parametric model for the cluster level clinical outcome and direct treatment effect on the surrogate can be used in place of proposed estimators, and this assumed structure of interference would replace assumption (e). We proceed with the above distribution free and finite population accommodating estimators for generality, although we do mention scenarios in Section 7 that will violate assumption (d) and therefore require modified cluster level estimators.

5.2. Population level estimation

Under Assumptions (a–e) one can specify models for the conditional estimands of interest and the predictiveness curves, i.e. use the true direct effect on the candidate surrogate as the regression covariate for the true potential outcomes at the cluster level, . For example, if one assumes the structural equation model for :

(5.2)

then

One could also assume, and then the total effect would be given by . As we never observe the true or the true potential clinical outcomes, this can be viewed as measurement error problem. When we can estimate the variance of , even conservatively, methods have been developed to account for these errors as well as the error in estimating (Cheng and Riu, 2006; Fuller, 1987).

Under the measurement error formulation, we can represent our available data, the estimates, as and where and have mean zero, which follows as the estimators are unbiased. However, given our setting, we cannot assume that the errors are the same over cluster or independent of each other within a cluster. To provide unbiased estimates of the desired coefficients under this type of measurement error under minimal assumptions we suggest the use of a modified version of the method of moments estimators, such as those given in Cheng and Riu (2006). As we allow for the possibility that the estimation errors of and are not independent, our suggested estimation method can account for this covariance provided it can be estimated. Further detail about our suggested estimation procedure, using a slightly modified version of the Cheng and Riu (2006) methods of moments estimator, is provided in the Section A of supplementary material available at Biostatistics online.

Although we select a simple linear formulation that allows for distribution free estimation of the estimands of interest for surrogate evaluation, any model that allows for a interaction between the direct effect on the candidate surrogate and the coverage level, or indicator of it, could be used. We select this method based on our desire to make limited assumptions about the within cluster data. However, this estimation could be undertaken in many different ways depending on the assumptions one is willing to make. Hierarchical Bayesian modeling, for example, could be used at the population or at the individual and cluster level combined. As mentioned above, if one is willing to make modeling assumptions about the within cluster data, a parametric model that matches these assumptions and accounts for the assumed structure of interference could be used. For example, a network model within cluster. Then estimation at the population level could be undertaken parametrically to account for measurement error.

When error estimates are not available, or if measurement error can be assumed ignorable, for example in very large cluster sizes, one could also consider a simple linear model based on the observed data. Here this is given by:

Then we can estimate the indirect effect of treatment conditional on the estimated direct effect on the candidate surrogate as: . Although this is technically not the conditional value of interest, when variance estimators of the cluster level estimates are not available or difficult to obtain this, or a similar model based on the estimates, may be the only options for evaluating evidence of surrogacy. We provide simulation results for this method, as well as our proposed measurement error method in Section 6.

5.3. Testable hypotheses

The testable hypotheses will depend on the assumed models and possibly the methods of estimation. For population average causal necessity for the indirect effect, the null hypothesis is . For this reason, a direct test of this null is generally not possible, and instead confidence intervals based on the causal contrast at is of interest. For example, if the confidence interval for covers 0 tightly, it is evidence of ACN for the indirect effect of vaccination. Under our assumed Model (5.2), this would be the confidence interval for . For one-sided strong average causal sufficiency of the indirect effect we require for all . Under Model (5.2) and under ACN, and without loss of generality that larger direct effects on the surrogate are better, the null is . When ACN does not hold, it is possible that one-sided strong population average causal sufficiency holds vacuously for a surrogate with no value because and . One could sequentially test the null and upon rejection test , for a rigorous test of 1SSACS that holds only when there is variation in the CEP curve. We instead suggest first testing for WEM.

The null for WEM for the indirect effect is given by for all . Under our assumed Model (5.2), this is equivalent to testing the null . When a different model is assumed, there will be a different parameterization of the null hypotheses that corresponds to the criteria. Given the model suggested above for , 1SSACS would have a null of , WEM a null of and ACN a null of .

6. Simulated application

There have been no phase III trials of TBV vaccines, or, to our knowledge, any cluster randomized trials comparing coverage levels that measure intermediate as well as clinical outcomes. We therefore use a similar surrogate measure as the a recent phase I TBV trial (Talaat and others, 2016) and generate data for a simulated application given the number of clusters proposed in Delrieu and others (2015) for a Phase III TBV trial. In Talaat and others (2016), the vaccine induced strong antibody and some standard membrane feeding assay (SMFA) responses, but other assays did not show strong or detectable vaccine effects. Deciding which of these assays should be used as a surrogate is difficult and may lead to the wrong vaccine candidate being taken to phase III trials.

The SMFA is a mosquito feeding assay which seeks to measure the transmission-blocking activity of antibodies. For the SMFA, multiple mosquitoes are fed blood meals mixed with a given test subject’s purified antibodies and lab cultured malaria parasites in the gametocyte stage. These mosquitoes are allowed to develop parasites prior to dissection so that the number of malaria parasites in the oocyst stage for each fed mosquito can be observed. The mean count of malaria parasites in the gut of a fed mosquito is then compared with a similar sample of mosquitoes that fed on the same blood and gametocyte mixture, less the antibodies, to estimate the transmission reduction activity (TRA) for a subject. The proportional difference in these means is called the TRA, as outlined in Talaat and others (2016); Swihart and others (2017). We consider a candidate surrogate to be some measure of vaccine activity similar to TRA.

Following Delrieu and others (2015), our simulations consider randomization of 32 clusters, of sizes ranging from 400 to 1000 subjects, to either 10% coverage or 90% coverage with independence between clusters. Subjects within cluster are randomized with equal probability to either 1:9 or 9:1, vaccine or placebo. We consider a malaria endemic but seasonal region, such as Mali, where high rates of transmission occur during the wet season and low or no transmission occurs during the dry season. If one vaccinated during the dry season at least one month after the end of the previous wet season, one could obtain blood samples from subjects prior to the beginning of the next wet season and after all subjects have been cleared of detectable parasites. Under this design, there is little chance of interference between subjects within cluster with regard to the candidate surrogate. Greater explanation of this hypothetical setting is given in the Section A of supplementary material available at Biostatistics online.

Following this hypothesized setting, we generate the data under Assumptions (a–e). The true direct effect on the cluster candidate surrogate in cluster , , is the mean of over all subjects in cluster , regardless of randomization. The average individual potential outcomes are normal with cluster specific means for and for . Here the are Beta(2,0.85), the are uniform(0.05, 0.25) and the are normal with mean 0 and standard deviation 0.01. We then select the subjects for which we will observe , which leaves the subjects for which we will observe . We then calculate using estimator (5.1). We calculate an error variance for the estimator following Hudgens and Halloran (2008).

In this example, our clinical outcome of interest is the count of incident malaria infections among the unvaccinated during the wet season following vaccination in each cluster. The individual outcomes , which under assumption (e) are equal to , are Poisson with scenario specific parameters, , where .

where is the same value used to generate . Let , then the three scenarios can be categorized as , and . These scenarios correspond to a useless surrogate, a candidate surrogate with limited predictive value and a candidate surrogate that fulfills one-sided strong average causal sufficiency and average causal necessity. We obtain by taking the average over the for the given cluster. Although none of our methods require an error variance estimate for , we use the standard sample moment covariance estimate of and as the cluster specific estimate of the covariance of estimation errors, which performs well in our scenarios where a small amount of zero mean individual random error drives the covariance of the measurement error.

We generate 5000 data sets to investigate bias, type one error rate and power in our setting of 32 clusters over the three scenarios. We compare two methods of estimation: a modified version of the method of moments estimator of Cheng and Riu (2006), outlined above and detailed in the supplementary material available at Biostatistics online, and a linear model fit directly on the estimated cluster level outcomes. Table 1 gives the results of the simulated examples from each of the three scenarios and the two different methods of estimation. Code used for the simulations is available at https://github.com/eegabriel/merror-surrogate commit 27bc43c.

Table 1.

Coefficient estimation for Scenarios A, B, and C using modified method of moment estimates and linear regression on estimates

Scenario A -
	Method of Moments
Average coefficient	1.10	0.80	0.00	0.00
Power/Type 1 error rate	1.00	0.92	0.04	0.04
	Linear regression
Average coefficient	1.10	0.80	0.00	0.00
Power/Type 1 error rate	1.00	1.00	0.01	0.04
Scenario B ---
	Method of moments
Average coefficient	1.10	0.20	0.10	0.20
Power/Type 1 error rate	1.00	0.14	0.12	0.08
	Linear regression
Average coefficient	1.10	0.21	0.09	0.18
Power/Type 1 error rate	1.00	0.60	0.04	0.25
Scenario C -
	Method of moments
Average coefficient	1.10	0.01	0.00	1.01
Power/Type 1 error rate	0.99	0.05	0.04	0.73
	Linear regression
Average coefficient	1.10	0.05	0.00	0.92
Power/Type 1 error rate	1.00	0.12	0.00	1.00

As can be seen in Table 1 in these scenarios the method of moments estimation has lower bias, but both sets of estimates are relatively unbiased. When a Beta distribution with a more extreme variance is used, causing increased error variance in the clinical outcome, the results are more similar and the method of moments does not perform as well. These additional results are provided in the Section B of supplementary material available at Biostatistics online. Although it is expected that the coefficients will be attenuated to the null when linear regression is run on the observed values with error, the heterogeneous measurement error we induce in both the outcome and the covariate seems to induce bias in either direction as well as type one error rate inflation. For example, there is inflated type one error rate for in Scenario C for the linear regression method, while in Scenarios A and B there are overly conservative results. The sandwich standard errors, as well as the estimation of the estimation error of , are conservative and lead to lower power to detect real effects in both Scenarios B and C. As the number of clusters increases so does power for the method of moments estimator; additional simulation results with increased number of clusters are provided in Section B of the supplementary material available at Biostatistics online. Of course, as the cluster size increases, the within cluster estimation error will decrease, and the results will become more similar between the two methods.

Table 2 shows that we have good power to detect WEM and 1SSACS when there is a high quality surrogate, such as in scenario C. However, when there is only minor violation of ACN, as in Scenario B, we see an inflated type one error rate for 1SSACS, as the confidence interval for ACN covers zero in this small number of clusters. This inflated type I error rate returns to nominal as the number of clusters increases; results provided in Section B of the supplementary material available at Biostatistics online.

Table 2.

Criteria testing/demonstrating using modified method of moment estimates

	ACN	1SSACS	WEM
	(95% CI Coverage of 0)	(power for if ACN not rejected)	(Power for )
Scenario A	0.08	0.01	0.04
Scenario B	0.86	0.07	0.08
Scenario C	0.95	0.71	0.73

7. Trial designs

If one assumes that indirect and total effects are monotone increasing with the difference between the two coverage levels, the most powerful trial design with two coverage levels would be to randomize to 100% and 0% coverage, as this allows for estimation of the largest total effect. For this reason, cluster randomized trials with 100% and 0% coverage are very common. Clearly, if the coverage levels were truly 100% and 0%, we would not be able to estimate , which is why we instead consider coverages near 100% and near 0%. The percentage of subjects that would ideally be randomized to placebo in the nearly 100% group that maximizes the power to evaluate WEM will depend on the variance of and and the size of the cluster.

For example, assume and both have variance 1 and are normal with means 2 and , respectively, where is uniform(1, 1.5) and that there is a linear relationship between coverage and outcome such that , where is normal(0,1). Further suppose there are 32 clusters each of size 100 with equal randomization to coverage or . Simulation based power calculations suggest the highest power for detecting WEM occurs at , % and % coverage. However, for estimating the marginal treatment effect on the clinical outcome, not conditional on , randomizing to % or lower and % or higher coverage is more powerful.

Therefore, the most powerful design for evaluating predictive cluster level surrogates may not be the most powerful trial design for detecting clinical efficacy. We believe that compromises should be made to accommodate surrogate evaluation, particularly in settings such as our motivating example, where the cost of running multiple cluster randomized trials is much higher than running fewer but larger trials. In the example above, power to detect a marginal treatment effect on the clinical outcome is reduced by less than 2% by moving from to , which increases power for detecting WEM by 27%. Currently, the vast majority of cluster randomized trials randomize to either 100% or 0% coverage, making our suggested surrogate evaluation methods impossible to apply. However, in practice, actual coverage levels may vary naturally, as not all subjects in a given area will consent.

Clusters assigned to 100% coverage may vary in realized coverage between 92% and 99%. Then, if the clusters which were assigned to 0% could be encouraged to coverage between 1% and 10%, our proposed definition of predictive cluster level surrogacy can be evaluated without sacrificing notable power to detect the marginal treatment effect. A different within cluster estimation method may be required to obtain unbiased results given the observational nature of realized coverage level. If the variation in realized coverage can be explained with measured baseline covariates, inverse probability weighted (IPW) within cluster estimators, such as those suggested in Tchetgen and VanderWeele (2012) and Perez-Heydrich and others (2014), could be used in place of our suggested within cluster estimators.

8. Discussion

Although there is similar work on mediation, to the knowledge of the authors, this is the first work towards evaluation of a surrogate under the causal-association paradigm in the presence of interference. In particular, this work is novel for its evaluation of surrogacy for clinical outcome contrasts of the indirect and total effects. To this end, we propose a definition of surrogacy at the cluster level, under which interference on the clinical outcome can exist within cluster and the causal-association paradigm of surrogate evaluation can still be applied easily. We outline our suggested criteria for surrogate evaluation in this setting, following those from the principal surrogate literature. We show how existing estimators and measurement error methods can be used to evaluate them in simulations based on our motivating setting of TBV in malaria.

Although we suggest and investigate the use of some existing estimation methods, any principled estimation method that is justifiable based on the assumptions can be used. The estimation can and should be tailored to the trial design and the assumptions that are likely to hold. Although we focus on indirect effects on the clinical outcome, in a different setting surrogates of the direct effect on the clinical outcome may be of interest. In this case, one could potentially apply one of the trial level surrogate evaluation methods at the cluster level, as given in Gabriel and others (2016) for example. Of course, because there is interference, the direct effect on the clinical outcome, and in some cases the effect on the surrogate outcome, will be coverage, or otherwise, dependent, which may complicate both the evaluation and use of a validated surrogate. This is a future area of research for the authors.

Although cluster randomized trials are most commonly randomized to 100% and 0% coverage levels, natural variation in the realized coverage level, as we outline in Section 7, can be used to apply our proposed surrogate evaluation methods without sacrificing power to detect a marginal total treatment effect on the clinical outcome. In this case, inverse probability of treatment weighted estimators within cluster could potentially be used to account for the semi-observational nature of the randomization. This is also an area of future research for the authors.

Funding

E. E. Gabriel was partially supported by Vetenskapsrådet [2016-01267 and 2017-01898].